2023-03-19 09:53:32,628 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427 2023-03-19 09:53:32,641 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-19 09:53:32,657 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-19 09:53:32,657 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/cluster_7a353873-d325-433e-e096-98958ab2209b, deleteOnExit=true 2023-03-19 09:53:32,658 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-19 09:53:32,658 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/test.cache.data in system properties and HBase conf 2023-03-19 09:53:32,659 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/hadoop.tmp.dir in system properties and HBase conf 2023-03-19 09:53:32,659 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/hadoop.log.dir in system properties and HBase conf 2023-03-19 09:53:32,660 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-19 09:53:32,660 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-19 09:53:32,660 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-19 09:53:32,782 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-19 09:53:33,204 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-19 09:53:33,210 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-19 09:53:33,210 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-19 09:53:33,211 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-19 09:53:33,211 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-19 09:53:33,212 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-19 09:53:33,212 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-19 09:53:33,213 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-19 09:53:33,213 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-19 09:53:33,214 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-19 09:53:33,214 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/nfs.dump.dir in system properties and HBase conf 2023-03-19 09:53:33,215 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/java.io.tmpdir in system properties and HBase conf 2023-03-19 09:53:33,215 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-19 09:53:33,215 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-19 09:53:33,216 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-19 09:53:33,814 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-19 09:53:33,819 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-19 09:53:34,121 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-19 09:53:34,303 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-19 09:53:34,323 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-19 09:53:34,364 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-19 09:53:34,396 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/java.io.tmpdir/Jetty_localhost_44391_hdfs____.o6768b/webapp 2023-03-19 09:53:34,547 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44391 2023-03-19 09:53:34,603 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-19 09:53:34,604 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-19 09:53:35,049 WARN [Listener at localhost/39255] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-19 09:53:35,206 WARN [Listener at localhost/39255] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-19 09:53:35,231 WARN [Listener at localhost/39255] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-19 09:53:35,239 INFO [Listener at localhost/39255] log.Slf4jLog(67): jetty-6.1.26 2023-03-19 09:53:35,250 INFO [Listener at localhost/39255] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/java.io.tmpdir/Jetty_localhost_40901_datanode____jl0r1h/webapp 2023-03-19 09:53:35,384 INFO [Listener at localhost/39255] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40901 2023-03-19 09:53:35,801 WARN [Listener at localhost/45961] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-19 09:53:36,296 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6bbf23b80b3ada5b: Processing first storage report for DS-59145e27-1e4f-4661-8bc4-d6da235f70ea from datanode 8ae7b5f6-b356-4e6a-a1cd-cdb73824920a 2023-03-19 09:53:36,298 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6bbf23b80b3ada5b: from storage DS-59145e27-1e4f-4661-8bc4-d6da235f70ea node DatanodeRegistration(127.0.0.1:39477, datanodeUuid=8ae7b5f6-b356-4e6a-a1cd-cdb73824920a, infoPort=42597, infoSecurePort=0, ipcPort=45961, storageInfo=lv=-57;cid=testClusterID;nsid=883070758;c=1679219613898), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-19 09:53:36,298 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6bbf23b80b3ada5b: Processing first storage report for DS-ac1c8f7c-fc0a-427d-afab-74090e34ff83 from datanode 8ae7b5f6-b356-4e6a-a1cd-cdb73824920a 2023-03-19 09:53:36,299 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6bbf23b80b3ada5b: from storage DS-ac1c8f7c-fc0a-427d-afab-74090e34ff83 node DatanodeRegistration(127.0.0.1:39477, datanodeUuid=8ae7b5f6-b356-4e6a-a1cd-cdb73824920a, infoPort=42597, infoSecurePort=0, ipcPort=45961, storageInfo=lv=-57;cid=testClusterID;nsid=883070758;c=1679219613898), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-19 09:53:36,341 DEBUG [Listener at localhost/45961] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427 2023-03-19 09:53:36,404 INFO [Listener at localhost/45961] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/cluster_7a353873-d325-433e-e096-98958ab2209b/zookeeper_0, clientPort=54245, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/cluster_7a353873-d325-433e-e096-98958ab2209b/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/cluster_7a353873-d325-433e-e096-98958ab2209b/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-19 09:53:36,418 INFO [Listener at localhost/45961] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54245 2023-03-19 09:53:36,427 INFO [Listener at localhost/45961] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-19 09:53:36,428 INFO [Listener at localhost/45961] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-19 09:53:37,113 INFO [Listener at localhost/45961] util.FSUtils(471): Created version file at hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505 with version=8 2023-03-19 09:53:37,114 INFO [Listener at localhost/45961] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/hbase-staging 2023-03-19 09:53:37,436 INFO [Listener at localhost/45961] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-19 09:53:38,077 INFO [Listener at localhost/45961] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-19 09:53:38,108 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-19 09:53:38,109 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-19 09:53:38,109 INFO [Listener at localhost/45961] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-19 09:53:38,109 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-19 09:53:38,109 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-19 09:53:38,250 INFO [Listener at localhost/45961] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-19 09:53:38,331 DEBUG [Listener at localhost/45961] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-19 09:53:38,424 INFO [Listener at localhost/45961] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44865 2023-03-19 09:53:38,434 INFO [Listener at localhost/45961] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-19 09:53:38,436 INFO [Listener at localhost/45961] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-19 09:53:38,457 INFO [Listener at localhost/45961] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44865 connecting to ZooKeeper ensemble=127.0.0.1:54245 2023-03-19 09:53:38,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:448650x0, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-19 09:53:38,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44865-0x100fc31d44a0000 connected 2023-03-19 09:53:38,531 DEBUG [Listener at localhost/45961] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-19 09:53:38,532 DEBUG [Listener at localhost/45961] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-19 09:53:38,535 DEBUG [Listener at localhost/45961] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-19 09:53:38,543 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44865 2023-03-19 09:53:38,543 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44865 2023-03-19 09:53:38,544 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44865 2023-03-19 09:53:38,544 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44865 2023-03-19 09:53:38,544 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44865 2023-03-19 09:53:38,551 INFO [Listener at localhost/45961] master.HMaster(444): hbase.rootdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505, hbase.cluster.distributed=false 2023-03-19 09:53:38,626 INFO [Listener at localhost/45961] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-19 09:53:38,626 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-19 09:53:38,626 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-19 09:53:38,626 INFO [Listener at localhost/45961] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-19 09:53:38,627 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-19 09:53:38,627 INFO [Listener at localhost/45961] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-19 09:53:38,631 INFO [Listener at localhost/45961] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-19 09:53:38,634 INFO [Listener at localhost/45961] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34669 2023-03-19 09:53:38,636 INFO [Listener at localhost/45961] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-19 09:53:38,642 DEBUG [Listener at localhost/45961] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-19 09:53:38,643 INFO [Listener at localhost/45961] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-19 09:53:38,646 INFO [Listener at localhost/45961] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-19 09:53:38,648 INFO [Listener at localhost/45961] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34669 connecting to ZooKeeper ensemble=127.0.0.1:54245 2023-03-19 09:53:38,651 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:346690x0, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-19 09:53:38,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34669-0x100fc31d44a0001 connected 2023-03-19 09:53:38,652 DEBUG [Listener at localhost/45961] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-19 09:53:38,654 DEBUG [Listener at localhost/45961] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-19 09:53:38,654 DEBUG [Listener at localhost/45961] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-19 09:53:38,656 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34669 2023-03-19 09:53:38,656 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34669 2023-03-19 09:53:38,657 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34669 2023-03-19 09:53:38,657 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34669 2023-03-19 09:53:38,658 DEBUG [Listener at localhost/45961] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34669 2023-03-19 09:53:38,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:38,671 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-19 09:53:38,673 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:38,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-19 09:53:38,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-19 09:53:38,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:38,694 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-19 09:53:38,696 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,44865,1679219617257 from backup master directory 2023-03-19 09:53:38,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-19 09:53:38,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:38,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-19 09:53:38,700 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-19 09:53:38,700 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:38,703 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-19 09:53:38,705 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-19 09:53:38,798 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/hbase.id with ID: ec33c293-b9ca-4852-aa90-2f0875ecc85a 2023-03-19 09:53:38,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-19 09:53:38,859 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:38,897 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7a5d5fe6 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:38,929 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5db5a859, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:38,951 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:53:38,953 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-19 09:53:38,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-19 09:53:38,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-19 09:53:38,972 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:53:38,976 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:53:38,977 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-19 09:53:39,009 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store-tmp 2023-03-19 09:53:39,036 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:39,036 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-19 09:53:39,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:53:39,036 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:53:39,036 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-19 09:53:39,037 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:53:39,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:53:39,037 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-19 09:53:39,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/WALs/jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:39,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44865%2C1679219617257, suffix=, logDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/WALs/jenkins-hbase4.apache.org,44865,1679219617257, archiveDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/oldWALs, maxLogs=10 2023-03-19 09:53:39,113 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39477,DS-59145e27-1e4f-4661-8bc4-d6da235f70ea,DISK] 2023-03-19 09:53:39,121 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:53:39,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/WALs/jenkins-hbase4.apache.org,44865,1679219617257/jenkins-hbase4.apache.org%2C44865%2C1679219617257.1679219619071 2023-03-19 09:53:39,182 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39477,DS-59145e27-1e4f-4661-8bc4-d6da235f70ea,DISK]] 2023-03-19 09:53:39,182 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:53:39,182 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:39,185 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-19 09:53:39,186 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-19 09:53:39,240 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-19 09:53:39,247 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-19 09:53:39,269 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-19 09:53:39,281 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:39,287 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-19 09:53:39,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-19 09:53:39,309 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-19 09:53:39,314 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:53:39,315 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71639764, jitterRate=0.06751567125320435}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-19 09:53:39,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-19 09:53:39,317 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-19 09:53:39,336 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-19 09:53:39,336 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-19 09:53:39,339 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-19 09:53:39,341 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-19 09:53:39,373 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-03-19 09:53:39,373 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-19 09:53:39,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-19 09:53:39,404 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-19 09:53:39,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-19 09:53:39,434 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-19 09:53:39,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-19 09:53:39,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-19 09:53:39,447 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-19 09:53:39,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:39,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-19 09:53:39,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-19 09:53:39,469 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-19 09:53:39,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-19 09:53:39,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-19 09:53:39,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:39,475 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,44865,1679219617257, sessionid=0x100fc31d44a0000, setting cluster-up flag (Was=false) 2023-03-19 09:53:39,489 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:39,500 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-19 09:53:39,502 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:39,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:39,514 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-19 09:53:39,515 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:39,518 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.hbase-snapshot/.tmp 2023-03-19 09:53:39,562 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(952): ClusterId : ec33c293-b9ca-4852-aa90-2f0875ecc85a 2023-03-19 09:53:39,568 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-19 09:53:39,578 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-19 09:53:39,578 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-19 09:53:39,581 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-19 09:53:39,582 DEBUG [RS:0;jenkins-hbase4:34669] zookeeper.ReadOnlyZKClient(139): Connect 0x643192bc to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:39,587 DEBUG [RS:0;jenkins-hbase4:34669] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af6704a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:39,588 DEBUG [RS:0;jenkins-hbase4:34669] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ec9f9cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-19 09:53:39,621 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34669 2023-03-19 09:53:39,625 INFO [RS:0;jenkins-hbase4:34669] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-19 09:53:39,625 INFO [RS:0;jenkins-hbase4:34669] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-19 09:53:39,625 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1023): About to register with Master. 2023-03-19 09:53:39,628 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,44865,1679219617257 with isa=jenkins-hbase4.apache.org/172.31.14.131:34669, startcode=1679219618625 2023-03-19 09:53:39,646 DEBUG [RS:0;jenkins-hbase4:34669] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-19 09:53:39,655 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-19 09:53:39,667 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-19 09:53:39,667 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-19 09:53:39,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-19 09:53:39,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-19 09:53:39,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-19 09:53:39,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-19 09:53:39,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,673 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679219649673 2023-03-19 09:53:39,676 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-19 09:53:39,681 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-19 09:53:39,682 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-19 09:53:39,688 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-19 09:53:39,689 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-19 09:53:39,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-19 09:53:39,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-19 09:53:39,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-19 09:53:39,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-19 09:53:39,700 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,703 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-19 09:53:39,705 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-19 09:53:39,705 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-19 09:53:39,708 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-19 09:53:39,708 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-19 09:53:39,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679219619710,5,FailOnTimeoutGroup] 2023-03-19 09:53:39,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679219619711,5,FailOnTimeoutGroup] 2023-03-19 09:53:39,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-19 09:53:39,712 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,713 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41409, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-19 09:53:39,713 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,733 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-19 09:53:39,734 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-19 09:53:39,734 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505 2023-03-19 09:53:39,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:39,757 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:39,760 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505 2023-03-19 09:53:39,760 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-19 09:53:39,761 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39255 2023-03-19 09:53:39,761 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-19 09:53:39,764 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/info 2023-03-19 09:53:39,765 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-19 09:53:39,767 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:39,767 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-19 09:53:39,768 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-19 09:53:39,769 DEBUG [RS:0;jenkins-hbase4:34669] zookeeper.ZKUtil(162): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:39,769 WARN [RS:0;jenkins-hbase4:34669] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-19 09:53:39,770 INFO [RS:0;jenkins-hbase4:34669] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-19 09:53:39,770 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/WALs/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:39,772 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/rep_barrier 2023-03-19 09:53:39,780 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-19 09:53:39,781 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:39,782 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-19 09:53:39,780 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34669,1679219618625] 2023-03-19 09:53:39,786 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/table 2023-03-19 09:53:39,786 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-19 09:53:39,788 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:39,791 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740 2023-03-19 09:53:39,792 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740 2023-03-19 09:53:39,793 DEBUG [RS:0;jenkins-hbase4:34669] zookeeper.ZKUtil(162): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:39,797 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:53:39,799 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-19 09:53:39,804 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:53:39,805 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60747007, jitterRate=-0.09479905664920807}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:53:39,805 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-19 09:53:39,805 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-19 09:53:39,805 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-19 09:53:39,805 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-19 09:53:39,806 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-19 09:53:39,806 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-19 09:53:39,806 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-19 09:53:39,807 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-19 09:53:39,807 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-19 09:53:39,812 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-19 09:53:39,813 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-19 09:53:39,817 INFO [RS:0;jenkins-hbase4:34669] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-19 09:53:39,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-19 09:53:39,835 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-19 09:53:39,838 INFO [RS:0;jenkins-hbase4:34669] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-19 09:53:39,838 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-19 09:53:39,846 INFO [RS:0;jenkins-hbase4:34669] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-19 09:53:39,846 INFO [RS:0;jenkins-hbase4:34669] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,847 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-19 09:53:39,854 INFO [RS:0;jenkins-hbase4:34669] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,854 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,855 DEBUG [RS:0;jenkins-hbase4:34669] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-19 09:53:39,856 INFO [RS:0;jenkins-hbase4:34669] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,856 INFO [RS:0;jenkins-hbase4:34669] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,857 INFO [RS:0;jenkins-hbase4:34669] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,873 INFO [RS:0;jenkins-hbase4:34669] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-19 09:53:39,875 INFO [RS:0;jenkins-hbase4:34669] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34669,1679219618625-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:39,890 INFO [RS:0;jenkins-hbase4:34669] regionserver.Replication(203): jenkins-hbase4.apache.org,34669,1679219618625 started 2023-03-19 09:53:39,890 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34669,1679219618625, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34669, sessionid=0x100fc31d44a0001 2023-03-19 09:53:39,891 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-19 09:53:39,891 DEBUG [RS:0;jenkins-hbase4:34669] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:39,891 DEBUG [RS:0;jenkins-hbase4:34669] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34669,1679219618625' 2023-03-19 09:53:39,891 DEBUG [RS:0;jenkins-hbase4:34669] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:39,892 DEBUG [RS:0;jenkins-hbase4:34669] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:39,892 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-19 09:53:39,892 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-19 09:53:39,892 DEBUG [RS:0;jenkins-hbase4:34669] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:39,893 DEBUG [RS:0;jenkins-hbase4:34669] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34669,1679219618625' 2023-03-19 09:53:39,893 DEBUG [RS:0;jenkins-hbase4:34669] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-19 09:53:39,893 DEBUG [RS:0;jenkins-hbase4:34669] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-19 09:53:39,894 DEBUG [RS:0;jenkins-hbase4:34669] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-19 09:53:39,894 INFO [RS:0;jenkins-hbase4:34669] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-19 09:53:39,894 INFO [RS:0;jenkins-hbase4:34669] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-19 09:53:39,991 DEBUG [jenkins-hbase4:44865] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-19 09:53:39,994 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34669,1679219618625, state=OPENING 2023-03-19 09:53:40,002 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-19 09:53:40,004 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:40,005 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-19 09:53:40,006 INFO [RS:0;jenkins-hbase4:34669] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34669%2C1679219618625, suffix=, logDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/WALs/jenkins-hbase4.apache.org,34669,1679219618625, archiveDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/oldWALs, maxLogs=32 2023-03-19 09:53:40,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:53:40,030 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39477,DS-59145e27-1e4f-4661-8bc4-d6da235f70ea,DISK] 2023-03-19 09:53:40,035 INFO [RS:0;jenkins-hbase4:34669] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/WALs/jenkins-hbase4.apache.org,34669,1679219618625/jenkins-hbase4.apache.org%2C34669%2C1679219618625.1679219620011 2023-03-19 09:53:40,036 DEBUG [RS:0;jenkins-hbase4:34669] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39477,DS-59145e27-1e4f-4661-8bc4-d6da235f70ea,DISK]] 2023-03-19 09:53:40,200 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:40,203 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-19 09:53:40,207 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36202, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-19 09:53:40,220 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-19 09:53:40,220 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-19 09:53:40,224 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34669%2C1679219618625.meta, suffix=.meta, logDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/WALs/jenkins-hbase4.apache.org,34669,1679219618625, archiveDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/oldWALs, maxLogs=32 2023-03-19 09:53:40,248 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39477,DS-59145e27-1e4f-4661-8bc4-d6da235f70ea,DISK] 2023-03-19 09:53:40,254 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/WALs/jenkins-hbase4.apache.org,34669,1679219618625/jenkins-hbase4.apache.org%2C34669%2C1679219618625.meta.1679219620227.meta 2023-03-19 09:53:40,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39477,DS-59145e27-1e4f-4661-8bc4-d6da235f70ea,DISK]] 2023-03-19 09:53:40,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:53:40,257 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-19 09:53:40,279 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-19 09:53:40,286 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-19 09:53:40,293 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-19 09:53:40,293 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:40,294 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-19 09:53:40,294 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-19 09:53:40,297 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-19 09:53:40,299 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/info 2023-03-19 09:53:40,299 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/info 2023-03-19 09:53:40,299 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-19 09:53:40,300 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:40,301 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-19 09:53:40,302 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/rep_barrier 2023-03-19 09:53:40,302 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/rep_barrier 2023-03-19 09:53:40,303 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-19 09:53:40,303 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:40,304 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-19 09:53:40,305 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/table 2023-03-19 09:53:40,305 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/table 2023-03-19 09:53:40,306 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-19 09:53:40,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:40,309 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740 2023-03-19 09:53:40,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740 2023-03-19 09:53:40,316 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:53:40,319 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-19 09:53:40,320 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66322670, jitterRate=-0.011715203523635864}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:53:40,321 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-19 09:53:40,331 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679219620191 2023-03-19 09:53:40,348 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-19 09:53:40,349 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-19 09:53:40,349 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34669,1679219618625, state=OPEN 2023-03-19 09:53:40,352 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-19 09:53:40,352 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-19 09:53:40,356 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-19 09:53:40,356 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34669,1679219618625 in 343 msec 2023-03-19 09:53:40,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-19 09:53:40,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 535 msec 2023-03-19 09:53:40,369 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 802 msec 2023-03-19 09:53:40,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679219620369, completionTime=-1 2023-03-19 09:53:40,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-19 09:53:40,370 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-03-19 09:53:40,434 DEBUG [hconnection-0x2d86bf75-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:40,438 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36204, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=1 2023-03-19 09:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679219680454 2023-03-19 09:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679219740454 2023-03-19 09:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 84 msec 2023-03-19 09:53:40,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44865,1679219617257-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:40,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44865,1679219617257-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:40,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44865,1679219617257-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:40,480 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:44865, period=300000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:40,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-19 09:53:40,487 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-19 09:53:40,496 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-19 09:53:40,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-19 09:53:40,509 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-19 09:53:40,511 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-19 09:53:40,514 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-19 09:53:40,536 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:40,538 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120 empty. 2023-03-19 09:53:40,539 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:40,539 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-19 09:53:40,959 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-19 09:53:40,962 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c405ddb2855a0ca7512046e9ce410120, NAME => 'hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp 2023-03-19 09:53:41,375 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:41,375 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c405ddb2855a0ca7512046e9ce410120, disabling compactions & flushes 2023-03-19 09:53:41,375 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:53:41,375 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:53:41,375 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. after waiting 0 ms 2023-03-19 09:53:41,375 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:53:41,376 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:53:41,376 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c405ddb2855a0ca7512046e9ce410120: 2023-03-19 09:53:41,380 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-19 09:53:41,396 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679219621384"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219621384"}]},"ts":"1679219621384"} 2023-03-19 09:53:41,425 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-19 09:53:41,427 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-19 09:53:41,432 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219621428"}]},"ts":"1679219621428"} 2023-03-19 09:53:41,437 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-19 09:53:41,445 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c405ddb2855a0ca7512046e9ce410120, ASSIGN}] 2023-03-19 09:53:41,448 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c405ddb2855a0ca7512046e9ce410120, ASSIGN 2023-03-19 09:53:41,450 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c405ddb2855a0ca7512046e9ce410120, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=false 2023-03-19 09:53:41,601 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c405ddb2855a0ca7512046e9ce410120, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:41,601 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679219621601"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219621601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219621601"}]},"ts":"1679219621601"} 2023-03-19 09:53:41,606 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c405ddb2855a0ca7512046e9ce410120, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:53:41,766 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:53:41,767 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c405ddb2855a0ca7512046e9ce410120, NAME => 'hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:53:41,768 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:41,768 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:41,768 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:41,769 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:41,773 INFO [StoreOpener-c405ddb2855a0ca7512046e9ce410120-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:41,776 DEBUG [StoreOpener-c405ddb2855a0ca7512046e9ce410120-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/info 2023-03-19 09:53:41,776 DEBUG [StoreOpener-c405ddb2855a0ca7512046e9ce410120-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/info 2023-03-19 09:53:41,776 INFO [StoreOpener-c405ddb2855a0ca7512046e9ce410120-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c405ddb2855a0ca7512046e9ce410120 columnFamilyName info 2023-03-19 09:53:41,777 INFO [StoreOpener-c405ddb2855a0ca7512046e9ce410120-1] regionserver.HStore(310): Store=c405ddb2855a0ca7512046e9ce410120/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:41,779 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:41,780 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:41,786 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:53:41,798 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:53:41,799 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c405ddb2855a0ca7512046e9ce410120; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74177005, jitterRate=0.10532350838184357}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-19 09:53:41,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c405ddb2855a0ca7512046e9ce410120: 2023-03-19 09:53:41,801 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120., pid=6, masterSystemTime=1679219621760 2023-03-19 09:53:41,805 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:53:41,805 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:53:41,807 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c405ddb2855a0ca7512046e9ce410120, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:41,807 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679219621806"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219621806"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219621806"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219621806"}]},"ts":"1679219621806"} 2023-03-19 09:53:41,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-19 09:53:41,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c405ddb2855a0ca7512046e9ce410120, server=jenkins-hbase4.apache.org,34669,1679219618625 in 206 msec 2023-03-19 09:53:41,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-19 09:53:41,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c405ddb2855a0ca7512046e9ce410120, ASSIGN in 371 msec 2023-03-19 09:53:41,821 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-19 09:53:41,821 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219621821"}]},"ts":"1679219621821"} 2023-03-19 09:53:41,824 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-19 09:53:41,829 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-19 09:53:41,832 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3300 sec 2023-03-19 09:53:41,912 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-19 09:53:41,914 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-19 09:53:41,914 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:53:41,963 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-19 09:53:41,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-19 09:53:41,987 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 36 msec 2023-03-19 09:53:41,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-19 09:53:42,015 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-19 09:53:42,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 21 msec 2023-03-19 09:53:42,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-19 09:53:42,039 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-19 09:53:42,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.339sec 2023-03-19 09:53:42,041 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-19 09:53:42,042 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-19 09:53:42,043 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-19 09:53:42,044 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44865,1679219617257-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-19 09:53:42,045 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44865,1679219617257-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-19 09:53:42,055 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-19 09:53:42,068 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:42,072 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:42,085 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:42,099 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36214, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:42,109 INFO [Listener at localhost/45961] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:53:42,137 INFO [Listener at localhost/45961] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=395, ProcessCount=169, AvailableMemoryMB=2525 2023-03-19 09:53:42,156 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-19 09:53:42,160 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34748, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-19 09:53:42,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:53:42,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:53:42,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-19 09:53:42,177 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-19 09:53:42,180 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-19 09:53:42,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-19 09:53:42,183 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,185 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1 empty. 2023-03-19 09:53:42,186 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,186 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:53:42,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-19 09:53:42,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-19 09:53:42,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-19 09:53:42,607 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:53:42,609 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7ef6718d22690da9d245ca8892594cd1, NAME => 'TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp 2023-03-19 09:53:42,621 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:42,621 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7ef6718d22690da9d245ca8892594cd1, disabling compactions & flushes 2023-03-19 09:53:42,621 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,621 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,621 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. after waiting 0 ms 2023-03-19 09:53:42,621 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,621 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,621 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:42,625 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-19 09:53:42,627 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219622627"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219622627"}]},"ts":"1679219622627"} 2023-03-19 09:53:42,630 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-19 09:53:42,631 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-19 09:53:42,632 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219622632"}]},"ts":"1679219622632"} 2023-03-19 09:53:42,634 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-19 09:53:42,640 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7ef6718d22690da9d245ca8892594cd1, ASSIGN}] 2023-03-19 09:53:42,642 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7ef6718d22690da9d245ca8892594cd1, ASSIGN 2023-03-19 09:53:42,644 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7ef6718d22690da9d245ca8892594cd1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=false 2023-03-19 09:53:42,795 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7ef6718d22690da9d245ca8892594cd1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:42,796 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219622795"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219622795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219622795"}]},"ts":"1679219622795"} 2023-03-19 09:53:42,799 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:53:42,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-19 09:53:42,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7ef6718d22690da9d245ca8892594cd1, NAME => 'TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:53:42,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:53:42,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,961 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,963 DEBUG [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A 2023-03-19 09:53:42,963 DEBUG [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A 2023-03-19 09:53:42,965 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:53:42,966 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ef6718d22690da9d245ca8892594cd1 columnFamilyName A 2023-03-19 09:53:42,967 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.HStore(310): Store=7ef6718d22690da9d245ca8892594cd1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:42,967 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,969 DEBUG [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B 2023-03-19 09:53:42,969 DEBUG [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B 2023-03-19 09:53:42,969 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:53:42,969 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ef6718d22690da9d245ca8892594cd1 columnFamilyName B 2023-03-19 09:53:42,970 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.HStore(310): Store=7ef6718d22690da9d245ca8892594cd1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:42,970 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,972 DEBUG [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C 2023-03-19 09:53:42,972 DEBUG [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C 2023-03-19 09:53:42,972 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:53:42,973 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7ef6718d22690da9d245ca8892594cd1 columnFamilyName C 2023-03-19 09:53:42,974 INFO [StoreOpener-7ef6718d22690da9d245ca8892594cd1-1] regionserver.HStore(310): Store=7ef6718d22690da9d245ca8892594cd1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:53:42,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:53:42,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:42,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:53:42,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7ef6718d22690da9d245ca8892594cd1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65431108, jitterRate=-0.02500051259994507}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:53:42,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:42,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., pid=11, masterSystemTime=1679219622952 2023-03-19 09:53:42,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:42,994 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7ef6718d22690da9d245ca8892594cd1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:42,994 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219622994"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219622994"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219622994"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219622994"}]},"ts":"1679219622994"} 2023-03-19 09:53:43,001 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-19 09:53:43,001 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 in 198 msec 2023-03-19 09:53:43,005 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-19 09:53:43,006 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7ef6718d22690da9d245ca8892594cd1, ASSIGN in 362 msec 2023-03-19 09:53:43,007 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-19 09:53:43,007 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219623007"}]},"ts":"1679219623007"} 2023-03-19 09:53:43,009 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-19 09:53:43,125 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-19 09:53:43,128 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 952 msec 2023-03-19 09:53:43,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-19 09:53:43,306 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-19 09:53:43,311 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,315 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,318 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,321 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36226, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,325 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-19 09:53:43,328 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34758, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-19 09:53:43,338 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,343 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,344 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,351 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,352 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,365 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,366 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,371 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,371 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,375 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,376 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,384 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,384 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,417 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,418 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,425 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,425 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:53:43,430 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:53:43,448 DEBUG [hconnection-0x1b518f7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,449 DEBUG [hconnection-0x79e1e112-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,452 DEBUG [hconnection-0x1a5dd2ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,454 DEBUG [hconnection-0x139a00b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,455 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32860, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,455 DEBUG [hconnection-0x4896b578-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,456 DEBUG [hconnection-0x1330534b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,459 DEBUG [hconnection-0x27e010-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,459 DEBUG [hconnection-0x3fdf621b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,460 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32872, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,460 DEBUG [hconnection-0x2a12d69-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:53:43,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:43,468 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32876, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,468 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32904, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,472 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32888, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,479 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32916, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,491 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32942, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,502 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32932, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,511 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32958, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:53:43,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:43,536 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:43,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:43,540 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:43,540 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:43,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:43,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:43,543 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:43,543 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:43,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:43,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:43,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:43,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:43,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:43,549 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:43,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:43,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:43,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:43,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:43,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:43,565 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:43,567 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:43,571 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-19 09:53:43,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:43,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:43,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:43,596 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:43,597 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:43,597 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:43,597 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:43,597 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:43,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:43,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219683738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:43,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219623669, max timestamp=1679219623736Num uniques 859; 2023-03-19 09:53:43,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-19 09:53:43,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219623669, max timestamp=1679219623742Num uniques 859; 2023-03-19 09:53:43,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:53:43,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219623669, max timestamp=1679219623741Num uniques 859; 2023-03-19 09:53:43,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:53:43,835 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/be31484c1af242e6a27dfddf811a28f2 2023-03-19 09:53:43,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219623854, max timestamp=1679219623869Num uniques 1009; 2023-03-19 09:53:43,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-19 09:53:43,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219623854, max timestamp=1679219623869Num uniques 1018; 2023-03-19 09:53:43,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-19 09:53:43,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219623854, max timestamp=1679219623870Num uniques 1018; 2023-03-19 09:53:43,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-19 09:53:43,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:43,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219683942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:43,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:43,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219683947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:43,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219683948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:43,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219623929, max timestamp=1679219623948Num uniques 1009; 2023-03-19 09:53:43,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-19 09:53:43,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219623929, max timestamp=1679219623948Num uniques 1009; 2023-03-19 09:53:43,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-19 09:53:43,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219623929, max timestamp=1679219623948Num uniques 1009; 2023-03-19 09:53:43,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-19 09:53:43,979 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/0971a8aeb95043b2b6296fbbed686d26 2023-03-19 09:53:44,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219684027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219684030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219684053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219684054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,084 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/727400a903fa44fbb903e50130528b4c 2023-03-19 09:53:44,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/be31484c1af242e6a27dfddf811a28f2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/be31484c1af242e6a27dfddf811a28f2 2023-03-19 09:53:44,109 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/be31484c1af242e6a27dfddf811a28f2, entries=150, sequenceid=11, filesize=11.6 K 2023-03-19 09:53:44,118 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/0971a8aeb95043b2b6296fbbed686d26 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0971a8aeb95043b2b6296fbbed686d26 2023-03-19 09:53:44,134 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0971a8aeb95043b2b6296fbbed686d26, entries=150, sequenceid=11, filesize=11.6 K 2023-03-19 09:53:44,138 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/727400a903fa44fbb903e50130528b4c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/727400a903fa44fbb903e50130528b4c 2023-03-19 09:53:44,152 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/727400a903fa44fbb903e50130528b4c, entries=150, sequenceid=11, filesize=11.6 K 2023-03-19 09:53:44,155 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7ef6718d22690da9d245ca8892594cd1 in 584ms, sequenceid=11, compaction requested=false 2023-03-19 09:53:44,158 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-19 09:53:44,160 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:44,160 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:44,165 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,167 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,168 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,168 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:44,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ffc081b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:44,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:44,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,172 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:44,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:44,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:44,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:44,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:44,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=140.89 KB heapSize=403.92 KB 2023-03-19 09:53:44,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:44,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:53:44,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:44,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:53:44,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:44,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:53:44,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,180 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,180 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:44,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:44,189 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:44,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:44,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:44,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:44,189 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:44,205 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,206 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,206 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,230 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:44,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,232 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,232 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,233 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:44,234 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:44,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219684239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219684260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219684260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219684265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219684265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,334 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:44,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:44,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219684370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219684384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219684386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:44,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:44,391 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:44,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,397 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:44,397 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:44,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,399 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:44,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:44,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:44,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:44,413 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,413 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:44,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31dc2125[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:44,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:44,417 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:44,418 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,418 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:44,418 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:44,418 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:44,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,422 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:44,428 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:44,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:44,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:44,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:44,428 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:44,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,455 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:44,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:44,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:44,458 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:44,458 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:44,558 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:44,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:44,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:44,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219684579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219684579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219684582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:44,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:44,591 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:44,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:44,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:44,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,595 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:44,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:44,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:44,597 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:44,597 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:44,610 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,611 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:44,611 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:44,611 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:44,616 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219684603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219684610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,621 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:44,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@639b1c78[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:44,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:44,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,628 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,628 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:44,628 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:44,628 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:44,629 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,629 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:44,633 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:44,643 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:44,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:44,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:44,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:44,643 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:44,646 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,646 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,646 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/ee08f8167d564a28bfdc3767566d9e3c 2023-03-19 09:53:44,660 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,660 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,660 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,660 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:44,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:44,660 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:44,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:44,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:44,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:44,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:44,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:44,780 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:44,780 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,782 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:44,782 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:44,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,784 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:44,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:44,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:44,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:44,787 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:44,787 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:44,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:44,788 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:44,788 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:44,789 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:44,789 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:44,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:44,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:44,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ed34da[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:44,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:44,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,800 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,801 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:44,801 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:44,801 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:44,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,809 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:44,815 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:44,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:44,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:44,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:44,815 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:44,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:44,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:44,835 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:44,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,836 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:44,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:44,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219684890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219684922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:44,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219684924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:44,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:44,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:44,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:44,947 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:44,947 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,948 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:44,948 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:44,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:44,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:44,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:44,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:44,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:44,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:44,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:44,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:44,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:44,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:44,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:44,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:44,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:44,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:44,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:44,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:44,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,975 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:44,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b9e508c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:44,979 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:44,979 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,981 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,981 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:44,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:44,982 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:44,982 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:44,982 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:44,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:44,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:44,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:44,992 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:44,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:44,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:44,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:44,992 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:44,995 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,995 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:44,995 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:44,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:44,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:45,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:45,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,013 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:45,013 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:45,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219685087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219685087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,113 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:45,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:45,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:45,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:45,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:45,128 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:45,128 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:45,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:45,130 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,130 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,132 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,132 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:45,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:45,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:45,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:45,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:45,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:45,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:45,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:45,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:45,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:45,141 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:45,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,145 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:45,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1b9dd3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:45,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:45,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,150 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:45,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:45,154 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/c9b697debfe142d192a68eba4e921070 2023-03-19 09:53:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:45,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:45,163 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:45,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:45,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:45,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:45,163 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:45,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,171 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,174 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,178 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,178 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,178 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:45,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:45,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,180 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,180 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:45,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:45,216 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/4b6ddfa05e604a1ebb14026680c4f28f 2023-03-19 09:53:45,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/ee08f8167d564a28bfdc3767566d9e3c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee08f8167d564a28bfdc3767566d9e3c 2023-03-19 09:53:45,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee08f8167d564a28bfdc3767566d9e3c, entries=200, sequenceid=83, filesize=13.9 K 2023-03-19 09:53:45,244 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/c9b697debfe142d192a68eba4e921070 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c9b697debfe142d192a68eba4e921070 2023-03-19 09:53:45,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c9b697debfe142d192a68eba4e921070, entries=150, sequenceid=83, filesize=11.6 K 2023-03-19 09:53:45,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/4b6ddfa05e604a1ebb14026680c4f28f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/4b6ddfa05e604a1ebb14026680c4f28f 2023-03-19 09:53:45,280 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:45,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:45,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:45,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:45,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:45,294 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:45,294 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:45,297 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:45,297 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,297 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:45,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:45,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:45,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:45,307 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:45,307 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:45,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:45,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:45,321 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,337 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:45,337 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:45,337 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/4b6ddfa05e604a1ebb14026680c4f28f, entries=150, sequenceid=83, filesize=11.6 K 2023-03-19 09:53:45,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:45,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:45,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:45,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:45,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~403.88 KB/413568, currentSize=46.96 KB/48090 for 7ef6718d22690da9d245ca8892594cd1 in 1172ms, sequenceid=83, compaction requested=false 2023-03-19 09:53:45,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,354 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:45,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d82214[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:45,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:45,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,357 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:45,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:45,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:45,367 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:45,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:45,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:45,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:45,367 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:45,370 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,370 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,380 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,380 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:45,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:45,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:45,381 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:45,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,381 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:45,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:45,405 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:53:45,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:45,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:45,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:45,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:45,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:45,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:45,481 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:45,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:45,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:45,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:45,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:45,489 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:45,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:45,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:45,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:45,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:45,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:45,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:45,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:45,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:45,512 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,512 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:45,512 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:45,512 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:45,514 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:45,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dbe8131[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:45,519 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:45,519 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,520 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:45,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:45,524 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:45,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:45,529 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:45,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:45,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:45,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:45,529 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:45,532 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,532 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,532 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,534 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219625501, max timestamp=1679219625528Num uniques 859; 2023-03-19 09:53:45,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:53:45,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219625501, max timestamp=1679219625528Num uniques 859; 2023-03-19 09:53:45,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:53:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,545 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,545 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,545 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:45,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:45,545 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,546 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:45,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,546 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:45,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219685570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219685571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219685573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,648 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:45,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:45,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:45,660 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:45,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:45,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:45,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,671 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:45,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:45,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:45,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:45,685 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,685 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:45,685 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:45,685 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:45,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219685684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:45,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219685685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219685685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34dda23a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:45,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:45,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:45,699 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,699 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:45,703 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:45,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:45,708 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:45,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:45,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:45,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:45,708 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:45,710 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,710 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,710 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,713 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,720 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,720 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,720 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,721 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,721 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,721 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,721 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,721 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,721 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,721 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:45,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:45,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:45,723 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:45,723 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:45,823 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:45,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:45,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:45,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:45,836 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:45,836 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,840 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:45,840 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:45,841 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,841 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,844 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:45,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:45,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:45,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:45,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:45,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/4b44cfeb12694a10ae603d2794c082e6 2023-03-19 09:53:45,857 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,857 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332f5477[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:45,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:45,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,870 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-19 09:53:45,872 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:45,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:45,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:45,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:45,881 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:45,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:45,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:45,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:45,882 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:45,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,889 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,892 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:45,892 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,892 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:45,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,907 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:45,908 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:45,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:45,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:45,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:45,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:45,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:45,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219685899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:45,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:45,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:45,912 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:45,912 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:45,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219685901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:45,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219685901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:45,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/c4c8488c8eed437b91f20c3287a02a49 2023-03-19 09:53:46,012 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:46,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:46,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:46,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:46,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:46,028 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:46,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,029 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:46,029 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:46,029 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,033 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,039 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/0b5a2a56cdad4842bf544664c4f17fc1 2023-03-19 09:53:46,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:46,059 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-19 09:53:46,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:46,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:46,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:46,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:46,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:46,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:46,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:46,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:46,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:46,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,063 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:46,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12461884[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:46,066 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:46,066 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,070 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-19 09:53:46,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:46,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:46,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/4b44cfeb12694a10ae603d2794c082e6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b44cfeb12694a10ae603d2794c082e6 2023-03-19 09:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:46,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219686094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:46,099 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:46,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:46,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:46,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:46,099 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:46,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b44cfeb12694a10ae603d2794c082e6, entries=150, sequenceid=94, filesize=11.6 K 2023-03-19 09:53:46,103 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,104 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219686103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:46,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:46,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/c4c8488c8eed437b91f20c3287a02a49 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c4c8488c8eed437b91f20c3287a02a49 2023-03-19 09:53:46,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,120 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:46,122 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c4c8488c8eed437b91f20c3287a02a49, entries=150, sequenceid=94, filesize=11.6 K 2023-03-19 09:53:46,144 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/0b5a2a56cdad4842bf544664c4f17fc1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0b5a2a56cdad4842bf544664c4f17fc1 2023-03-19 09:53:46,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0b5a2a56cdad4842bf544664c4f17fc1, entries=150, sequenceid=94, filesize=11.6 K 2023-03-19 09:53:46,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 7ef6718d22690da9d245ca8892594cd1 in 760ms, sequenceid=94, compaction requested=true 2023-03-19 09:53:46,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:46,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:46,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:46,170 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:46,170 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:46,170 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:46,180 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:46,182 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:46,183 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,183 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0971a8aeb95043b2b6296fbbed686d26, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c9b697debfe142d192a68eba4e921070, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c4c8488c8eed437b91f20c3287a02a49] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=34.8 K 2023-03-19 09:53:46,187 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:46,187 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:46,187 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,187 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/be31484c1af242e6a27dfddf811a28f2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee08f8167d564a28bfdc3767566d9e3c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b44cfeb12694a10ae603d2794c082e6] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=37.1 K 2023-03-19 09:53:46,188 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0971a8aeb95043b2b6296fbbed686d26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679219623529 2023-03-19 09:53:46,188 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting be31484c1af242e6a27dfddf811a28f2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679219623529 2023-03-19 09:53:46,195 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c9b697debfe142d192a68eba4e921070, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219624015 2023-03-19 09:53:46,199 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ee08f8167d564a28bfdc3767566d9e3c, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219624013 2023-03-19 09:53:46,199 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c4c8488c8eed437b91f20c3287a02a49, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679219624225 2023-03-19 09:53:46,200 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 4b44cfeb12694a10ae603d2794c082e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679219624225 2023-03-19 09:53:46,222 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:46,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:46,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:46,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:46,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:46,252 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:46,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:46,257 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=131.94 KB heapSize=383.31 KB 2023-03-19 09:53:46,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:46,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:53:46,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:46,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:53:46,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:46,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:53:46,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:46,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:46,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:46,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:46,284 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#9 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:46,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:46,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:46,303 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:46,303 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:46,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:46,307 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#11 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b48b4f4[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:46,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:46,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,314 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:46,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:46,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:46,316 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:46,320 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:46,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:46,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:46,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:46,320 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:46,323 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,329 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:46,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219686309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,333 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,333 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,338 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:46,338 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:46,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,338 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219686314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219686317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:46,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/ec82845d21f84092bdbeddcc679c98cc 2023-03-19 09:53:46,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/4828c7eb5c58409fb87eb69010409e2e 2023-03-19 09:53:46,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:46,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:46,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219686440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219686441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:46,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:46,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:46,448 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:46,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:46,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:46,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219686446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,455 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:46,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:46,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:46,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:46,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:46,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:46,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725d0fd5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:46,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:46,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:46,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:46,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/e8f2f0272cfa4299bb2e3765ac39417e 2023-03-19 09:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:46,513 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:46,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:46,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:46,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:46,513 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:46,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,518 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/ec82845d21f84092bdbeddcc679c98cc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ec82845d21f84092bdbeddcc679c98cc 2023-03-19 09:53:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,527 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:46,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:46,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:46,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:46,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:46,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ec82845d21f84092bdbeddcc679c98cc, entries=200, sequenceid=126, filesize=13.9 K 2023-03-19 09:53:46,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/4828c7eb5c58409fb87eb69010409e2e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4828c7eb5c58409fb87eb69010409e2e 2023-03-19 09:53:46,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4828c7eb5c58409fb87eb69010409e2e, entries=150, sequenceid=126, filesize=11.6 K 2023-03-19 09:53:46,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/e8f2f0272cfa4299bb2e3765ac39417e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e8f2f0272cfa4299bb2e3765ac39417e 2023-03-19 09:53:46,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e8f2f0272cfa4299bb2e3765ac39417e, entries=150, sequenceid=126, filesize=11.6 K 2023-03-19 09:53:46,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~383.27 KB/392464, currentSize=53.67 KB/54960 for 7ef6718d22690da9d245ca8892594cd1 in 323ms, sequenceid=126, compaction requested=true 2023-03-19 09:53:46,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:46,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-19 09:53:46,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-19 09:53:46,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-19 09:53:46,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:46,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:46,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:46,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:46,644 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:46,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:46,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:46,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,652 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,653 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,653 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,656 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:53:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:46,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:46,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:46,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:46,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:46,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:46,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:46,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:46,665 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:46,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:46,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:46,671 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:46,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:46,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:46,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b987105[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:46,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:46,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,682 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:46,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:46,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:46,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:46,685 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:46,691 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:46,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:46,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:46,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:46,692 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:46,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,699 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,707 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,707 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:46,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:46,711 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:46,713 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:46,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219686753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219686753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219686753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,802 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/94be782b596d48afa07b99d7b0b87abb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/94be782b596d48afa07b99d7b0b87abb 2023-03-19 09:53:46,808 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/472ec6fa1a4b424087a0616c21d98c4f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/472ec6fa1a4b424087a0616c21d98c4f 2023-03-19 09:53:46,813 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:46,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:46,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:46,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:46,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:46,822 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:46,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,824 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:46,824 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:46,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,826 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,827 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,827 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:46,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:46,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:46,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:46,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:46,833 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into 94be782b596d48afa07b99d7b0b87abb(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:46,833 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,833 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=13, startTime=1679219626169; duration=0sec 2023-03-19 09:53:46,833 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 472ec6fa1a4b424087a0616c21d98c4f(size=11.7 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:46,833 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:46,833 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=13, startTime=1679219626166; duration=0sec 2023-03-19 09:53:46,834 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-19 09:53:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:46,834 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-19 09:53:46,834 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:46,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:46,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b8dcca[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:46,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:46,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,849 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:46,850 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:53:46,850 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-03-19 09:53:46,850 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,850 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:46,852 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:46,852 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:46,852 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-19 09:53:46,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,853 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:46,853 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/727400a903fa44fbb903e50130528b4c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/4b6ddfa05e604a1ebb14026680c4f28f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0b5a2a56cdad4842bf544664c4f17fc1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e8f2f0272cfa4299bb2e3765ac39417e] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=46.4 K 2023-03-19 09:53:46,853 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:46,853 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:46,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,854 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:46,855 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 727400a903fa44fbb903e50130528b4c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679219623529 2023-03-19 09:53:46,855 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:46,855 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:46,855 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:46,855 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,855 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-19 09:53:46,856 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,856 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:46,855 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4b6ddfa05e604a1ebb14026680c4f28f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219624015 2023-03-19 09:53:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,858 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0b5a2a56cdad4842bf544664c4f17fc1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679219624225 2023-03-19 09:53:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,858 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:46,858 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:46,858 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,858 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e8f2f0272cfa4299bb2e3765ac39417e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679219625556 2023-03-19 09:53:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:46,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:46,862 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:46,863 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:46,863 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:46,863 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:46,863 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:46,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219686860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,865 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,865 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,865 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:46,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219686861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:46,866 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219686862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:46,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:46,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:46,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:46,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:46,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:46,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:46,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:46,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,877 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:46,877 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:46,900 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#15 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:46,937 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/34c38bdc6cb946818f8f57bb89d20dd5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/34c38bdc6cb946818f8f57bb89d20dd5 2023-03-19 09:53:46,951 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into 34c38bdc6cb946818f8f57bb89d20dd5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:46,952 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:46,952 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=12, startTime=1679219626581; duration=0sec 2023-03-19 09:53:46,952 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:46,977 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:46,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:46,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:46,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:46,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:46,984 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:46,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,988 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:46,988 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:46,988 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,988 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:46,991 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:46,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:46,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:46,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:46,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:46,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:46,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:46,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:46,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:46,996 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,996 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:46,996 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:46,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:46,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:46,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:46,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:46,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:46,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:46,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,000 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,001 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,001 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:47,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7739241d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:47,004 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:47,004 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:47,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:47,009 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:47,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:47,014 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:47,014 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:47,014 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:47,015 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:47,015 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:47,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:47,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,029 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:47,029 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:47,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:47,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:47,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:47,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:47,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:47,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:47,030 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:47,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,030 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:47,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219687068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219687068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219687069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/20ad5c89fd6a4877948480b571dc6dd4 2023-03-19 09:53:47,132 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:47,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:47,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:47,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:47,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:47,141 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:47,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:47,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:47,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:47,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,145 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:47,145 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:47,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:47,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:47,156 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:47,157 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:47,159 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:47,159 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:47,159 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:47,159 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:47,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/2d587ae5dfd9474cb9e225ff2b65ccca 2023-03-19 09:53:47,162 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,166 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,167 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,167 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:47,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d00ae88[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:47,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:47,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,174 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:47,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:47,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:47,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:47,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:47,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:47,176 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:47,181 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:47,181 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:47,181 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:47,181 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:47,184 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,185 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:47,185 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/7038ff2871364e1aacbc4d4d0855c506 2023-03-19 09:53:47,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:47,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:47,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:47,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:47,200 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:47,200 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:47,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,200 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:47,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:47,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/20ad5c89fd6a4877948480b571dc6dd4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/20ad5c89fd6a4877948480b571dc6dd4 2023-03-19 09:53:47,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/20ad5c89fd6a4877948480b571dc6dd4, entries=150, sequenceid=139, filesize=11.8 K 2023-03-19 09:53:47,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/2d587ae5dfd9474cb9e225ff2b65ccca as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/2d587ae5dfd9474cb9e225ff2b65ccca 2023-03-19 09:53:47,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/2d587ae5dfd9474cb9e225ff2b65ccca, entries=150, sequenceid=139, filesize=11.8 K 2023-03-19 09:53:47,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/7038ff2871364e1aacbc4d4d0855c506 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7038ff2871364e1aacbc4d4d0855c506 2023-03-19 09:53:47,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7038ff2871364e1aacbc4d4d0855c506, entries=150, sequenceid=139, filesize=11.8 K 2023-03-19 09:53:47,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 7ef6718d22690da9d245ca8892594cd1 in 593ms, sequenceid=139, compaction requested=true 2023-03-19 09:53:47,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:47,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:47,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:47,249 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:47,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:47,249 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:47,251 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35911 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:47,251 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:47,251 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:47,251 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/94be782b596d48afa07b99d7b0b87abb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4828c7eb5c58409fb87eb69010409e2e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/2d587ae5dfd9474cb9e225ff2b65ccca] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=35.1 K 2023-03-19 09:53:47,255 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38251 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:47,255 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:47,255 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:47,255 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/472ec6fa1a4b424087a0616c21d98c4f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ec82845d21f84092bdbeddcc679c98cc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/20ad5c89fd6a4877948480b571dc6dd4] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=37.4 K 2023-03-19 09:53:47,256 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 94be782b596d48afa07b99d7b0b87abb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679219624225 2023-03-19 09:53:47,257 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 472ec6fa1a4b424087a0616c21d98c4f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679219624225 2023-03-19 09:53:47,269 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 4828c7eb5c58409fb87eb69010409e2e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679219625556 2023-03-19 09:53:47,275 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ec82845d21f84092bdbeddcc679c98cc, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679219625556 2023-03-19 09:53:47,276 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2d587ae5dfd9474cb9e225ff2b65ccca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679219626287 2023-03-19 09:53:47,277 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 20ad5c89fd6a4877948480b571dc6dd4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679219626287 2023-03-19 09:53:47,294 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:47,300 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:47,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:47,303 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#19 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:47,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:47,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:47,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:47,310 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:47,310 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:47,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:47,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:47,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,314 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:47,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:47,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:47,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:47,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:47,322 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:47,322 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:47,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:47,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:47,334 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:47,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:47,335 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:47,335 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-19 09:53:47,340 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:47,341 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:47,341 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:47,341 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:47,341 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:47,343 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:47,374 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/708c1a3c3139478e991400a65a3ba6dc 2023-03-19 09:53:47,375 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/978224aad8bf435eb2430b7363daabcf as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/978224aad8bf435eb2430b7363daabcf 2023-03-19 09:53:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:47,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:47,392 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 978224aad8bf435eb2430b7363daabcf(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:47,392 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:47,392 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=13, startTime=1679219627249; duration=0sec 2023-03-19 09:53:47,392 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:47,392 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-19 09:53:47,398 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:47,398 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:47,398 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:47,413 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/37e5b9e205724c4496785bada70a7d39 2023-03-19 09:53:47,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219687412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219687413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219687416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,433 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-19 09:53:47,434 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-19 09:53:47,487 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/382312e6aa9142fc89e7554e38bae254 2023-03-19 09:53:47,520 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/708c1a3c3139478e991400a65a3ba6dc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/708c1a3c3139478e991400a65a3ba6dc 2023-03-19 09:53:47,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219687521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,528 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/708c1a3c3139478e991400a65a3ba6dc, entries=150, sequenceid=162, filesize=11.8 K 2023-03-19 09:53:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219687524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219687525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,533 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/37e5b9e205724c4496785bada70a7d39 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/37e5b9e205724c4496785bada70a7d39 2023-03-19 09:53:47,543 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/37e5b9e205724c4496785bada70a7d39, entries=150, sequenceid=162, filesize=11.8 K 2023-03-19 09:53:47,546 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/382312e6aa9142fc89e7554e38bae254 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/382312e6aa9142fc89e7554e38bae254 2023-03-19 09:53:47,555 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/382312e6aa9142fc89e7554e38bae254, entries=150, sequenceid=162, filesize=11.8 K 2023-03-19 09:53:47,558 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 7ef6718d22690da9d245ca8892594cd1 in 223ms, sequenceid=162, compaction requested=true 2023-03-19 09:53:47,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:47,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:47,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:47,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:47,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:47,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:47,564 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:47,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1568ea74[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:47,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:47,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,568 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:47,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:47,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:47,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:47,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:47,575 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:47,575 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:47,575 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:47,575 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:47,575 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:47,585 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,585 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:47,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,585 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:47,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,586 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:47,589 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:47,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:47,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:47,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:47,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:47,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:47,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:47,592 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:47,593 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,593 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:47,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:47,593 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:47,693 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:47,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:47,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:47,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:47,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:47,700 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:47,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:47,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:47,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:47,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,704 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:47,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:47,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:47,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:47,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:47,708 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:47,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:47,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:47,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:47,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:47,711 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:47,711 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:47,711 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:53:47,711 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:47,711 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:47,711 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:47,712 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:47,712 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:47,712 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:47,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:47,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:47,755 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/95ded8d2b1b34c1eb87bf504fbc397c1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/95ded8d2b1b34c1eb87bf504fbc397c1 2023-03-19 09:53:47,768 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into 95ded8d2b1b34c1eb87bf504fbc397c1(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:47,768 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:47,768 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=13, startTime=1679219627249; duration=0sec 2023-03-19 09:53:47,768 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:47,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219687771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219687773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219687773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219687876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219687886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:47,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:47,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219687887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219688080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219688088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219688092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219688098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,100 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:53:48,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219688115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,117 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:53:48,146 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/964364d3f3f24513b1bf45455f2827ed 2023-03-19 09:53:48,186 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/7ee62272798b476f97cf192da313aa6a 2023-03-19 09:53:48,224 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/eb1b0e215ebb4ae2819ab27b01f47fc5 2023-03-19 09:53:48,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/964364d3f3f24513b1bf45455f2827ed as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/964364d3f3f24513b1bf45455f2827ed 2023-03-19 09:53:48,248 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/964364d3f3f24513b1bf45455f2827ed, entries=150, sequenceid=176, filesize=11.8 K 2023-03-19 09:53:48,251 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/7ee62272798b476f97cf192da313aa6a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/7ee62272798b476f97cf192da313aa6a 2023-03-19 09:53:48,260 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/7ee62272798b476f97cf192da313aa6a, entries=150, sequenceid=176, filesize=11.8 K 2023-03-19 09:53:48,264 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/eb1b0e215ebb4ae2819ab27b01f47fc5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/eb1b0e215ebb4ae2819ab27b01f47fc5 2023-03-19 09:53:48,274 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/eb1b0e215ebb4ae2819ab27b01f47fc5, entries=150, sequenceid=176, filesize=11.8 K 2023-03-19 09:53:48,275 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7ef6718d22690da9d245ca8892594cd1 in 564ms, sequenceid=176, compaction requested=true 2023-03-19 09:53:48,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:48,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:48,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,291 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae93a9f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:48,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:48,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:48,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:48,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,299 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:48,304 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:48,304 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:48,304 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:48,304 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:48,304 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:48,307 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,307 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,321 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,321 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:48,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:48,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,328 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:48,329 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:48,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-19 09:53:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:48,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219688412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219688415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219688417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,433 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:48,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:48,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:48,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:48,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:48,451 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:48,451 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:48,455 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:48,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/7c59af2a4301416eb9dad5081125bee5 2023-03-19 09:53:48,455 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,458 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:48,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:48,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:48,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:48,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:48,466 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,466 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:48,466 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:48,466 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:48,471 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744ef613[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:48,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:48,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:48,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:48,489 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:48,499 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:48,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:48,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:48,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:48,500 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:48,503 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,503 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,503 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,506 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:48,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:48,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:48,509 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:48,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219688520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219688521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/4fd8ffe13974406b871e291467124e91 2023-03-19 09:53:48,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219688523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/63c993707c034e6a8befcb051dcb4812 2023-03-19 09:53:48,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/7c59af2a4301416eb9dad5081125bee5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7c59af2a4301416eb9dad5081125bee5 2023-03-19 09:53:48,609 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:48,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:48,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7c59af2a4301416eb9dad5081125bee5, entries=200, sequenceid=197, filesize=14.1 K 2023-03-19 09:53:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:48,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/4fd8ffe13974406b871e291467124e91 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4fd8ffe13974406b871e291467124e91 2023-03-19 09:53:48,617 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:48,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:48,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:48,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:48,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:48,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:48,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4fd8ffe13974406b871e291467124e91, entries=150, sequenceid=197, filesize=11.8 K 2023-03-19 09:53:48,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/63c993707c034e6a8befcb051dcb4812 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/63c993707c034e6a8befcb051dcb4812 2023-03-19 09:53:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:48,636 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,637 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:48,637 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:48,637 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:48,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c5262fc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:48,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:48,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/63c993707c034e6a8befcb051dcb4812, entries=150, sequenceid=197, filesize=11.8 K 2023-03-19 09:53:48,648 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:48,648 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,648 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:48,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 7ef6718d22690da9d245ca8892594cd1 in 261ms, sequenceid=197, compaction requested=true 2023-03-19 09:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:48,649 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:48,649 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:48,650 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,650 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,652 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,652 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:48,652 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,652 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50740 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:48,652 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/95ded8d2b1b34c1eb87bf504fbc397c1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/37e5b9e205724c4496785bada70a7d39, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/7ee62272798b476f97cf192da313aa6a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4fd8ffe13974406b871e291467124e91] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=47.2 K 2023-03-19 09:53:48,652 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:48,652 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,653 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/978224aad8bf435eb2430b7363daabcf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/708c1a3c3139478e991400a65a3ba6dc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/964364d3f3f24513b1bf45455f2827ed, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7c59af2a4301416eb9dad5081125bee5] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=49.6 K 2023-03-19 09:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,654 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 95ded8d2b1b34c1eb87bf504fbc397c1, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679219626287 2023-03-19 09:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,655 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 978224aad8bf435eb2430b7363daabcf, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679219626287 2023-03-19 09:53:48,655 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 37e5b9e205724c4496785bada70a7d39, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679219626705 2023-03-19 09:53:48,656 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 708c1a3c3139478e991400a65a3ba6dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679219626705 2023-03-19 09:53:48,656 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 7ee62272798b476f97cf192da313aa6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679219627398 2023-03-19 09:53:48,657 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4fd8ffe13974406b871e291467124e91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219627769 2023-03-19 09:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,658 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 964364d3f3f24513b1bf45455f2827ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679219627398 2023-03-19 09:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,659 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7c59af2a4301416eb9dad5081125bee5, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219627769 2023-03-19 09:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:48,660 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:48,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:48,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:48,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:48,660 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:48,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,666 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:48,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,678 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:48,678 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:48,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,682 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#29 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:48,691 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:48,733 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-19 09:53:48,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:48,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:48,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:48,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/382630546d1e445aac160c34686008da 2023-03-19 09:53:48,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219688760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219688761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219688765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,779 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:48,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:48,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:48,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:48,784 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:48,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:48,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:48,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,786 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:48,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:48,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:48,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:48,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/0843f42708344af29ce3f77cdd871579 2023-03-19 09:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:48,789 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,789 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:48,789 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:48,789 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,790 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:48,790 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,792 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:48,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e3b1048[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:48,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:48,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,800 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:48,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:48,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:48,807 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:48,807 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:48,808 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:48,808 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:48,808 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:48,810 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,810 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,810 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:48,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:48,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,817 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:48,817 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:48,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/d7c90852426c4872a04d4d877c41f01a 2023-03-19 09:53:48,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/382630546d1e445aac160c34686008da as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/382630546d1e445aac160c34686008da 2023-03-19 09:53:48,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/382630546d1e445aac160c34686008da, entries=150, sequenceid=212, filesize=11.8 K 2023-03-19 09:53:48,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/0843f42708344af29ce3f77cdd871579 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0843f42708344af29ce3f77cdd871579 2023-03-19 09:53:48,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0843f42708344af29ce3f77cdd871579, entries=150, sequenceid=212, filesize=11.8 K 2023-03-19 09:53:48,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/d7c90852426c4872a04d4d877c41f01a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/d7c90852426c4872a04d4d877c41f01a 2023-03-19 09:53:48,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/d7c90852426c4872a04d4d877c41f01a, entries=150, sequenceid=212, filesize=11.8 K 2023-03-19 09:53:48,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7ef6718d22690da9d245ca8892594cd1 in 137ms, sequenceid=212, compaction requested=true 2023-03-19 09:53:48,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:48,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-19 09:53:48,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-19 09:53:48,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-19 09:53:48,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:48,873 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:53:48,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:48,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,874 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:48,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,874 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:48,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:48,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219688894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219688896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:48,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219688896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,917 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:48,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:48,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:48,922 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:48,922 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,922 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:48,922 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:48,923 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,923 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,924 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,924 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:48,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:48,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:48,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:48,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:48,930 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,930 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,930 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,931 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d85e6d2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:48,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:48,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,937 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:48,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:48,939 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:48,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:48,943 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:48,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:48,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:48,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:48,943 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:48,945 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,945 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,945 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,946 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:48,948 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:48,953 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:48,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:48,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:48,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:48,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:48,953 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:48,953 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:48,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:48,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:48,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219688999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219689000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219689000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,053 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:49,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:49,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:49,061 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:49,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:49,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:49,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,063 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,063 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,064 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,064 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:49,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:49,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:49,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:49,068 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,068 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:49,068 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:49,068 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:49,070 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,070 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,071 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,071 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:49,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e786f1f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:49,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:49,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,075 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:49,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:49,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:49,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:49,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:49,082 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:49,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:49,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:49,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:49,083 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:49,087 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,087 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,087 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,101 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:49,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:49,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,102 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:49,102 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:49,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:49,134 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/0f4e155b34eb4804ab724838a6e2131a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0f4e155b34eb4804ab724838a6e2131a 2023-03-19 09:53:49,139 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/8aac5b07628a46849cbe916fa0fb1c06 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8aac5b07628a46849cbe916fa0fb1c06 2023-03-19 09:53:49,143 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into 0f4e155b34eb4804ab724838a6e2131a(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:49,143 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:49,143 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=12, startTime=1679219628649; duration=0sec 2023-03-19 09:53:49,144 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-19 09:53:49,144 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-19 09:53:49,154 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 8aac5b07628a46849cbe916fa0fb1c06(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:49,154 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:49,154 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=12, startTime=1679219628649; duration=0sec 2023-03-19 09:53:49,154 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-19 09:53:49,155 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72204 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-19 09:53:49,155 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:53:49,155 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 6 compacting, 0 eligible, 16 blocking 2023-03-19 09:53:49,155 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,155 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:49,155 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:49,156 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:49,156 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-19 09:53:49,156 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/34c38bdc6cb946818f8f57bb89d20dd5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7038ff2871364e1aacbc4d4d0855c506, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/382312e6aa9142fc89e7554e38bae254, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/eb1b0e215ebb4ae2819ab27b01f47fc5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/63c993707c034e6a8befcb051dcb4812, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/d7c90852426c4872a04d4d877c41f01a] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=70.5 K 2023-03-19 09:53:49,157 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 34c38bdc6cb946818f8f57bb89d20dd5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679219625556 2023-03-19 09:53:49,157 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:49,158 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:49,158 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:49,158 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-19 09:53:49,158 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 7038ff2871364e1aacbc4d4d0855c506, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679219626287 2023-03-19 09:53:49,159 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 382312e6aa9142fc89e7554e38bae254, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679219626705 2023-03-19 09:53:49,159 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:49,159 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:49,159 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:49,159 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting eb1b0e215ebb4ae2819ab27b01f47fc5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679219627398 2023-03-19 09:53:49,160 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 63c993707c034e6a8befcb051dcb4812, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219627769 2023-03-19 09:53:49,161 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d7c90852426c4872a04d4d877c41f01a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679219628413 2023-03-19 09:53:49,179 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:49,192 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/3e5272e7a8934e1d850f3b9c8bc22eaa as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/3e5272e7a8934e1d850f3b9c8bc22eaa 2023-03-19 09:53:49,202 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:49,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219689203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:49,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219689203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,205 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into 3e5272e7a8934e1d850f3b9c8bc22eaa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:49,205 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:49,205 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=10, startTime=1679219628871; duration=0sec 2023-03-19 09:53:49,205 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:49,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:49,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:49,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:49,209 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:49,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:49,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:49,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219689205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:49,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:49,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:49,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:49,219 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,219 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:49,220 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:49,220 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:49,222 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,222 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:49,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41b2b1ae[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:49,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:49,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:49,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:49,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:49,239 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,239 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:49,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:49,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:49,246 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:49,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:49,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:49,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:49,246 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:49,253 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,253 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,253 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,255 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,255 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,259 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,259 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,259 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,259 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:49,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:49,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,268 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:49,268 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:49,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/1da3b6f8126840a4b7de43fe705aea4d 2023-03-19 09:53:49,368 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:49,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:49,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:49,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:49,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:49,373 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:49,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:49,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:49,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,376 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,376 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,376 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,376 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:49,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:49,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:49,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:49,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:49,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:49,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:49,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:49,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,380 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579d5080[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:49,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:49,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,384 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:49,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,385 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:49,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:49,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:49,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:49,389 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:49,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:49,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:49,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:49,389 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:49,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:49,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:49,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,398 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:49,398 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:49,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,498 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:49,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:49,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:49,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:49,504 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:49,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:49,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:49,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219689505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:49,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:49,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:49,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:49,510 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,510 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:49,510 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:49,510 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:49,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219689508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,512 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219689513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:49,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35cc8fd7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:49,515 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:49,515 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:49,517 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:49,517 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:49,517 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,517 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:49,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:49,522 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:49,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:49,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:49,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:49,522 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:49,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,525 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,527 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:49,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:49,533 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:49,533 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:49,633 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:49,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:49,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:49,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:49,656 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:49,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:49,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:49,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,658 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:49,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:49,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:49,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:49,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:49,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:49,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:49,666 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:49,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c11ee6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:49,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:49,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,666 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:49,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:49,670 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,670 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:49,671 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:49,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:49,674 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:49,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:49,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:49,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:49,674 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:49,675 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,676 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,676 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:49,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:49,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:49,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,690 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:49,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:49,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,690 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,690 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:49,690 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:49,690 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:49,690 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:49,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/058fdd26bd4f4e2fb01656028d3f1c0e 2023-03-19 09:53:49,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/7cebd238c36e4e168bf44430894b7e1a 2023-03-19 09:53:49,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/1da3b6f8126840a4b7de43fe705aea4d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1da3b6f8126840a4b7de43fe705aea4d 2023-03-19 09:53:49,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1da3b6f8126840a4b7de43fe705aea4d, entries=150, sequenceid=230, filesize=11.8 K 2023-03-19 09:53:49,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/058fdd26bd4f4e2fb01656028d3f1c0e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/058fdd26bd4f4e2fb01656028d3f1c0e 2023-03-19 09:53:49,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/058fdd26bd4f4e2fb01656028d3f1c0e, entries=150, sequenceid=230, filesize=11.8 K 2023-03-19 09:53:49,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/7cebd238c36e4e168bf44430894b7e1a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7cebd238c36e4e168bf44430894b7e1a 2023-03-19 09:53:49,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7cebd238c36e4e168bf44430894b7e1a, entries=150, sequenceid=230, filesize=11.8 K 2023-03-19 09:53:49,789 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7ef6718d22690da9d245ca8892594cd1 in 916ms, sequenceid=230, compaction requested=true 2023-03-19 09:53:49,789 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:49,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:49,790 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:49,790 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-19 09:53:49,792 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:49,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:49,793 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:49,793 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:49,793 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,794 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8aac5b07628a46849cbe916fa0fb1c06, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/382630546d1e445aac160c34686008da, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1da3b6f8126840a4b7de43fe705aea4d] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=35.6 K 2023-03-19 09:53:49,794 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:49,795 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:49,795 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,795 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0f4e155b34eb4804ab724838a6e2131a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0843f42708344af29ce3f77cdd871579, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/058fdd26bd4f4e2fb01656028d3f1c0e] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=35.6 K 2023-03-19 09:53:49,795 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0f4e155b34eb4804ab724838a6e2131a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219627769 2023-03-19 09:53:49,796 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8aac5b07628a46849cbe916fa0fb1c06, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219627769 2023-03-19 09:53:49,797 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0843f42708344af29ce3f77cdd871579, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679219628413 2023-03-19 09:53:49,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:49,798 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 382630546d1e445aac160c34686008da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679219628413 2023-03-19 09:53:49,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:49,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:49,799 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:49,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:49,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:49,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,800 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1da3b6f8126840a4b7de43fe705aea4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219628754 2023-03-19 09:53:49,801 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 058fdd26bd4f4e2fb01656028d3f1c0e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219628754 2023-03-19 09:53:49,801 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:49,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:49,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:49,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:49,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:49,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:49,804 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:49,804 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:49,804 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:49,804 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:49,804 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-19 09:53:49,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:49,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:49,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:49,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:49,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:49,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:49,813 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:49,817 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:50,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:50,009 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:50,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219690038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219690039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219690041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219690146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219690146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219690146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,228 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/8cef5e5fabb94cb3baeef22108b314fd 2023-03-19 09:53:50,247 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/3f519429470f4843983b917635f4a9a7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3f519429470f4843983b917635f4a9a7 2023-03-19 09:53:50,258 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/d5c7bbc01f3844cb99b887183b9c1d93 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/d5c7bbc01f3844cb99b887183b9c1d93 2023-03-19 09:53:50,259 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 3f519429470f4843983b917635f4a9a7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:50,259 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:50,259 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=13, startTime=1679219629790; duration=0sec 2023-03-19 09:53:50,260 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:50,260 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-19 09:53:50,261 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:53:50,262 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:53:50,262 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:53:50,266 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into d5c7bbc01f3844cb99b887183b9c1d93(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:50,266 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:50,266 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=13, startTime=1679219629791; duration=0sec 2023-03-19 09:53:50,267 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:50,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219690348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219690351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219690352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32932 deadline: 1679219690652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,656 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/5f727232a9c54179aafd72a8fbab253f 2023-03-19 09:53:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219690655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:50,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32888 deadline: 1679219690656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,075 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/e4f14638552e49129c2542e14b8bf6e4 2023-03-19 09:53:51,083 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/8cef5e5fabb94cb3baeef22108b314fd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8cef5e5fabb94cb3baeef22108b314fd 2023-03-19 09:53:51,094 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8cef5e5fabb94cb3baeef22108b314fd, entries=150, sequenceid=247, filesize=11.8 K 2023-03-19 09:53:51,096 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/5f727232a9c54179aafd72a8fbab253f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5f727232a9c54179aafd72a8fbab253f 2023-03-19 09:53:51,106 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5f727232a9c54179aafd72a8fbab253f, entries=150, sequenceid=247, filesize=11.8 K 2023-03-19 09:53:51,107 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/e4f14638552e49129c2542e14b8bf6e4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e4f14638552e49129c2542e14b8bf6e4 2023-03-19 09:53:51,114 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e4f14638552e49129c2542e14b8bf6e4, entries=150, sequenceid=247, filesize=11.8 K 2023-03-19 09:53:51,115 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7ef6718d22690da9d245ca8892594cd1 in 1311ms, sequenceid=247, compaction requested=true 2023-03-19 09:53:51,115 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:51,115 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:51,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c2ae791[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:51,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:51,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:51,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,123 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:51,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:51,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:51,129 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:51,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:51,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:51,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:51,129 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:51,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,133 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,139 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,142 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:51,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:51,142 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,142 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,142 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:51,142 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:51,142 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,142 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:51,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-19 09:53:51,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:51,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:51,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:51,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:51,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:51,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:51,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/560ebda97f4445b2bceba5ef80717dad 2023-03-19 09:53:51,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219691216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219691216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219691221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,243 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:51,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:51,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:51,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:51,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:51,258 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:51,258 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,258 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:51,258 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:51,259 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,259 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,261 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:51,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:51,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:51,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:51,262 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,262 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:51,263 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:51,263 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:51,267 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,267 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,269 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,269 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:51,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:51,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510eb3c4[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:51,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:51,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:51,276 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:51,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:51,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:51,279 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:51,285 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:51,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:51,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:51,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:51,285 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:51,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,292 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,295 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,295 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,295 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,295 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:51,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:51,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,295 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:51,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:51,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:51,295 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,296 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,296 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,296 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219691322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219691322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219691326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,396 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:51,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:51,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:51,401 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:51,401 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,401 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:51,401 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:51,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,403 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,403 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:51,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:51,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:51,407 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:51,407 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:51,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:51,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:51,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:51,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:51,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:51,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed2c05[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:51,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:51,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,421 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,423 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:51,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:51,426 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:51,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:51,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:51,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:51,427 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:51,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:51,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:51,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,436 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,436 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,436 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:51,436 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:51,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:51,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219691526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219691527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32872 deadline: 1679219691528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,536 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:51,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:51,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:51,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:51,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:51,541 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:51,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:51,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:51,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:51,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:51,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:51,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:51,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:51,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:51,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:51,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30eb24d0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:51,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:51,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:51,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:51,557 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:51,560 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:51,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:51,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:51,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:51,560 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:51,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:51,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,570 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:51,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:51,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:51,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:51,573 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:51,573 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:51,574 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:51,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:51,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/725b5dbcc05c47cca7a3d6b617833e68 2023-03-19 09:53:51,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/dc0d7e5799ed427eb57ab23678424f63 2023-03-19 09:53:51,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/560ebda97f4445b2bceba5ef80717dad as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/560ebda97f4445b2bceba5ef80717dad 2023-03-19 09:53:51,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/560ebda97f4445b2bceba5ef80717dad, entries=150, sequenceid=268, filesize=11.9 K 2023-03-19 09:53:51,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/725b5dbcc05c47cca7a3d6b617833e68 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/725b5dbcc05c47cca7a3d6b617833e68 2023-03-19 09:53:51,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/725b5dbcc05c47cca7a3d6b617833e68, entries=150, sequenceid=268, filesize=11.9 K 2023-03-19 09:53:51,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/dc0d7e5799ed427eb57ab23678424f63 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/dc0d7e5799ed427eb57ab23678424f63 2023-03-19 09:53:51,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/dc0d7e5799ed427eb57ab23678424f63, entries=150, sequenceid=268, filesize=11.9 K 2023-03-19 09:53:51,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 7ef6718d22690da9d245ca8892594cd1 in 490ms, sequenceid=268, compaction requested=true 2023-03-19 09:53:51,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:51,655 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:51,657 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:51,657 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:51,657 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:51,657 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:51,657 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:51,657 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:51,657 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,657 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3f519429470f4843983b917635f4a9a7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8cef5e5fabb94cb3baeef22108b314fd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/560ebda97f4445b2bceba5ef80717dad] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=35.8 K 2023-03-19 09:53:51,658 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3f519429470f4843983b917635f4a9a7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219628754 2023-03-19 09:53:51,659 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:51,665 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:51,665 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,665 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/d5c7bbc01f3844cb99b887183b9c1d93, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5f727232a9c54179aafd72a8fbab253f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/725b5dbcc05c47cca7a3d6b617833e68] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=35.8 K 2023-03-19 09:53:51,659 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8cef5e5fabb94cb3baeef22108b314fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679219628886 2023-03-19 09:53:51,666 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d5c7bbc01f3844cb99b887183b9c1d93, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219628754 2023-03-19 09:53:51,667 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 560ebda97f4445b2bceba5ef80717dad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219630032 2023-03-19 09:53:51,667 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5f727232a9c54179aafd72a8fbab253f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679219628886 2023-03-19 09:53:51,668 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 725b5dbcc05c47cca7a3d6b617833e68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219630032 2023-03-19 09:53:51,674 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:51,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:51,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:51,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:51,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:51,680 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:51,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:51,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:51,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,684 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:51,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:51,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:51,701 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:51,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:51,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:51,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:51,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:51,725 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:51,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:51,733 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:51,733 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:51,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:51,733 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:53:51,733 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:51,733 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:51,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:51,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:51,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:51,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:51,761 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/c2c1ef10a24f40ffba3be1b04d060146 2023-03-19 09:53:51,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:51,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:51,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219691865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219691867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219691868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219691973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219691975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219691978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219692127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,137 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:53:52,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219692138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,147 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:53:52,150 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/8b13177fa0ef4322886e079f6520168f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8b13177fa0ef4322886e079f6520168f 2023-03-19 09:53:52,158 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/bcd50886789649e89bf26a7f0781005e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/bcd50886789649e89bf26a7f0781005e 2023-03-19 09:53:52,159 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 8b13177fa0ef4322886e079f6520168f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:52,159 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:52,159 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=13, startTime=1679219631655; duration=0sec 2023-03-19 09:53:52,159 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:52,159 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:52,161 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48636 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:52,162 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:53:52,162 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:52,162 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/3e5272e7a8934e1d850f3b9c8bc22eaa, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7cebd238c36e4e168bf44430894b7e1a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e4f14638552e49129c2542e14b8bf6e4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/dc0d7e5799ed427eb57ab23678424f63] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=47.5 K 2023-03-19 09:53:52,163 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3e5272e7a8934e1d850f3b9c8bc22eaa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679219628413 2023-03-19 09:53:52,164 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7cebd238c36e4e168bf44430894b7e1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219628754 2023-03-19 09:53:52,165 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting e4f14638552e49129c2542e14b8bf6e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679219628886 2023-03-19 09:53:52,166 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting dc0d7e5799ed427eb57ab23678424f63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219630032 2023-03-19 09:53:52,170 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into bcd50886789649e89bf26a7f0781005e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:52,171 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:52,172 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=13, startTime=1679219631657; duration=0sec 2023-03-19 09:53:52,172 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:52,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219692176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219692178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,181 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#50 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:52,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219692182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,188 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/75ee3cb4a6c14065989e2814fcf1f9f2 2023-03-19 09:53:52,200 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/818a1d5cd1c54969ace69a1fcc7778ab as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/818a1d5cd1c54969ace69a1fcc7778ab 2023-03-19 09:53:52,207 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into 818a1d5cd1c54969ace69a1fcc7778ab(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:52,207 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:52,207 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=12, startTime=1679219631657; duration=0sec 2023-03-19 09:53:52,207 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:52,216 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/69d08d9508af4536939a1f2d21239c3c 2023-03-19 09:53:52,223 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/c2c1ef10a24f40ffba3be1b04d060146 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c2c1ef10a24f40ffba3be1b04d060146 2023-03-19 09:53:52,231 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c2c1ef10a24f40ffba3be1b04d060146, entries=150, sequenceid=283, filesize=11.9 K 2023-03-19 09:53:52,232 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/75ee3cb4a6c14065989e2814fcf1f9f2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/75ee3cb4a6c14065989e2814fcf1f9f2 2023-03-19 09:53:52,238 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/75ee3cb4a6c14065989e2814fcf1f9f2, entries=150, sequenceid=283, filesize=11.9 K 2023-03-19 09:53:52,239 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/69d08d9508af4536939a1f2d21239c3c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/69d08d9508af4536939a1f2d21239c3c 2023-03-19 09:53:52,269 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/69d08d9508af4536939a1f2d21239c3c, entries=150, sequenceid=283, filesize=11.9 K 2023-03-19 09:53:52,272 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 7ef6718d22690da9d245ca8892594cd1 in 539ms, sequenceid=283, compaction requested=false 2023-03-19 09:53:52,273 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:52,273 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:52,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:52,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:52,286 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b464b33[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:52,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:52,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:52,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:52,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:52,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:52,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:52,300 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:52,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:52,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:52,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:52,300 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:52,302 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,302 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,302 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:52,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:52,303 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,308 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:52,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:52,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:52,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:52,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:52,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:52,312 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:52,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:52,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:52,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:52,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:52,312 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:52,412 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:52,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:52,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:52,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:52,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:52,425 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:52,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:52,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:52,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:52,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:52,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:52,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:52,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:52,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:52,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:52,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:52,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:52,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:52,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:52,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:52,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:52,436 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:53:52,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:52,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:52,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:52,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:52,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:52,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:52,457 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/9a001e35c8d5430697b7b18c5bd115e9 2023-03-19 09:53:52,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:52,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:52,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219692499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219692500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219692504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219692603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219692604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219692609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219692807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219692807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:52,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219692817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:52,881 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/9a0b813f66684cdd9783981c44cbf367 2023-03-19 09:53:52,949 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/9984520baca44d5f8ef0264f0a4fa5fc 2023-03-19 09:53:52,962 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/9a001e35c8d5430697b7b18c5bd115e9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9a001e35c8d5430697b7b18c5bd115e9 2023-03-19 09:53:52,971 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9a001e35c8d5430697b7b18c5bd115e9, entries=150, sequenceid=304, filesize=11.9 K 2023-03-19 09:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/9a0b813f66684cdd9783981c44cbf367 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/9a0b813f66684cdd9783981c44cbf367 2023-03-19 09:53:52,981 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/9a0b813f66684cdd9783981c44cbf367, entries=150, sequenceid=304, filesize=11.9 K 2023-03-19 09:53:52,990 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/9984520baca44d5f8ef0264f0a4fa5fc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/9984520baca44d5f8ef0264f0a4fa5fc 2023-03-19 09:53:53,007 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/9984520baca44d5f8ef0264f0a4fa5fc, entries=150, sequenceid=304, filesize=11.9 K 2023-03-19 09:53:53,009 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7ef6718d22690da9d245ca8892594cd1 in 573ms, sequenceid=304, compaction requested=true 2023-03-19 09:53:53,009 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:53,009 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:53,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,016 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f72490c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:53,022 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:53,022 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,023 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:53,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:53,063 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:53,073 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:53,073 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:53,073 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:53,073 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:53,073 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:53,080 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,080 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,080 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,083 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:53,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:53,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:53,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:53,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,087 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,087 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,087 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:53,087 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:53,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,086 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:53,118 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:53:53,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:53,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:53,120 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:53,120 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:53,120 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:53,120 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:53,120 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:53,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219693154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219693153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219693157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/b1fe2338b4ba4cce802928a022969e9f 2023-03-19 09:53:53,187 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:53,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:53,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:53,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:53,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:53,202 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:53,202 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:53,202 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:53,202 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:53,202 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,203 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,204 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:53,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:53,204 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:53,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:53,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:53,211 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:53,211 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:53,211 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:53,211 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:53,214 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,214 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:53,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4616d5e3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:53,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:53,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:53,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:53,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:53,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:53,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:53,229 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:53,229 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:53,229 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:53,229 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:53,229 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:53,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:53,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:53,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:53,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:53,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,240 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:53,240 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:53,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219693260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219693261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219693261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,341 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:53,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:53,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:53,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:53,345 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:53,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:53,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:53,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:53,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:53,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:53,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:53,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:53,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:53,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:53,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:53,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:53,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f32b455[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:53,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:53,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:53,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:53,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:53,368 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,369 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:53,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:53,380 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:53,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:53,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:53,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:53,380 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:53,387 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,388 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,387 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,394 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,397 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:53,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:53,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:53,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:53,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,398 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:53,398 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:53,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:53,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219693467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219693467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219693469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:53,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:53,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:53,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:53,503 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:53,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:53,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:53,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:53,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,545 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,545 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:53,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:53,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:53,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:53,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:53,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:53,547 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:53,547 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:53,547 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:53,547 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:53,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:53,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:53,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,575 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,575 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,575 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16146d53[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:53,577 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:53,578 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,597 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/a9c09061f4074a32a0b914aef88af512 2023-03-19 09:53:53,633 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:53,635 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,635 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:53,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:53,672 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:53,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:53,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:53,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:53,672 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:53,745 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,745 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:53,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,745 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:53,746 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:53,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219693772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219693773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:53,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219693774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:53,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,960 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:53,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:53,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:53,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:53,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:53,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:53,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:53,961 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:53,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:53,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:53,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:53,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/747e7f7d63e44ac2a0e19935c97909e6 2023-03-19 09:53:54,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/b1fe2338b4ba4cce802928a022969e9f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/b1fe2338b4ba4cce802928a022969e9f 2023-03-19 09:53:54,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/b1fe2338b4ba4cce802928a022969e9f, entries=200, sequenceid=320, filesize=14.3 K 2023-03-19 09:53:54,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/a9c09061f4074a32a0b914aef88af512 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/a9c09061f4074a32a0b914aef88af512 2023-03-19 09:53:54,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/a9c09061f4074a32a0b914aef88af512, entries=150, sequenceid=320, filesize=11.9 K 2023-03-19 09:53:54,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/747e7f7d63e44ac2a0e19935c97909e6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/747e7f7d63e44ac2a0e19935c97909e6 2023-03-19 09:53:54,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/747e7f7d63e44ac2a0e19935c97909e6, entries=150, sequenceid=320, filesize=11.9 K 2023-03-19 09:53:54,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 7ef6718d22690da9d245ca8892594cd1 in 926ms, sequenceid=320, compaction requested=true 2023-03-19 09:53:54,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:54,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:54,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:54,045 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:54,045 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:54,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:54,047 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:54,047 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:54,047 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51730 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:54,048 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,048 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:54,048 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/bcd50886789649e89bf26a7f0781005e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/75ee3cb4a6c14065989e2814fcf1f9f2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/9a0b813f66684cdd9783981c44cbf367, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/a9c09061f4074a32a0b914aef88af512] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=48.1 K 2023-03-19 09:53:54,048 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,048 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8b13177fa0ef4322886e079f6520168f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c2c1ef10a24f40ffba3be1b04d060146, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9a001e35c8d5430697b7b18c5bd115e9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/b1fe2338b4ba4cce802928a022969e9f] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=50.5 K 2023-03-19 09:53:54,048 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting bcd50886789649e89bf26a7f0781005e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219630032 2023-03-19 09:53:54,049 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8b13177fa0ef4322886e079f6520168f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219630032 2023-03-19 09:53:54,049 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 75ee3cb4a6c14065989e2814fcf1f9f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679219631214 2023-03-19 09:53:54,049 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c2c1ef10a24f40ffba3be1b04d060146, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679219631214 2023-03-19 09:53:54,060 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9a001e35c8d5430697b7b18c5bd115e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679219631851 2023-03-19 09:53:54,060 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9a0b813f66684cdd9783981c44cbf367, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679219631851 2023-03-19 09:53:54,061 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b1fe2338b4ba4cce802928a022969e9f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679219632490 2023-03-19 09:53:54,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:54,061 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a9c09061f4074a32a0b914aef88af512, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679219632490 2023-03-19 09:53:54,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:54,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:54,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:54,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:54,068 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:54,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:54,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:54,069 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:54,069 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,069 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,077 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#58 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:54,080 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#59 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:54,094 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/4b3e21b0627747b88a52fa86e39cbca2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b3e21b0627747b88a52fa86e39cbca2 2023-03-19 09:53:54,098 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:54,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:54,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:54,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:54,102 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,102 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:54,102 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:54,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:54,105 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 4b3e21b0627747b88a52fa86e39cbca2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:54,105 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:54,105 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=12, startTime=1679219634045; duration=0sec 2023-03-19 09:53:54,105 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:54,105 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:54,107 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:54,107 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:53:54,107 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,107 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/818a1d5cd1c54969ace69a1fcc7778ab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/69d08d9508af4536939a1f2d21239c3c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/9984520baca44d5f8ef0264f0a4fa5fc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/747e7f7d63e44ac2a0e19935c97909e6] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=48.1 K 2023-03-19 09:53:54,108 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 818a1d5cd1c54969ace69a1fcc7778ab, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219630032 2023-03-19 09:53:54,108 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 69d08d9508af4536939a1f2d21239c3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679219631214 2023-03-19 09:53:54,109 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9984520baca44d5f8ef0264f0a4fa5fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679219631851 2023-03-19 09:53:54,109 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 747e7f7d63e44ac2a0e19935c97909e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679219632490 2023-03-19 09:53:54,122 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/5a7aa9ceda304597a364349b1b25ee67 2023-03-19 09:53:54,122 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#61 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:54,144 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/071d60ce5ebc4c74875c18545fce22af as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/071d60ce5ebc4c74875c18545fce22af 2023-03-19 09:53:54,152 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into 071d60ce5ebc4c74875c18545fce22af(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:54,152 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:54,152 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=12, startTime=1679219634045; duration=0sec 2023-03-19 09:53:54,152 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:54,153 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/eb07496d2d5945c589abecc962812408 2023-03-19 09:53:54,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:54,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:54,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219694299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219694300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219694305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219694405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219694406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219694409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,499 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/b0e5394317d24e58878913b878bc2bff as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b0e5394317d24e58878913b878bc2bff 2023-03-19 09:53:54,505 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into b0e5394317d24e58878913b878bc2bff(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:54,505 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:54,505 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=12, startTime=1679219634045; duration=0sec 2023-03-19 09:53:54,506 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:54,573 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/ccf5803a8c444ec3b3cf8b0714e3f013 2023-03-19 09:53:54,579 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/5a7aa9ceda304597a364349b1b25ee67 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/5a7aa9ceda304597a364349b1b25ee67 2023-03-19 09:53:54,589 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/5a7aa9ceda304597a364349b1b25ee67, entries=150, sequenceid=336, filesize=11.9 K 2023-03-19 09:53:54,590 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/eb07496d2d5945c589abecc962812408 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/eb07496d2d5945c589abecc962812408 2023-03-19 09:53:54,597 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/eb07496d2d5945c589abecc962812408, entries=150, sequenceid=336, filesize=11.9 K 2023-03-19 09:53:54,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/ccf5803a8c444ec3b3cf8b0714e3f013 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/ccf5803a8c444ec3b3cf8b0714e3f013 2023-03-19 09:53:54,609 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/ccf5803a8c444ec3b3cf8b0714e3f013, entries=150, sequenceid=336, filesize=11.9 K 2023-03-19 09:53:54,610 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 7ef6718d22690da9d245ca8892594cd1 in 508ms, sequenceid=336, compaction requested=false 2023-03-19 09:53:54,610 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:54,610 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:54,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b136cb4[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:54,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:54,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,620 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:54,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:54,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:54,623 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:54,624 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:54,627 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:54,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:54,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:54,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:54,627 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:54,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:54,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,638 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:54,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:54,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,643 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,643 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:54,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:54,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:54,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:54,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:54,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:54,643 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:54,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219694653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219694654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219694655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/3b88cfa0384e42389821fa728f03b177 2023-03-19 09:53:54,743 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:54,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:54,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:54,755 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:54,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:54,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:54,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:54,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,757 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:54,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:54,757 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:54,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:54,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219694759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:54,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219694760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219694760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:54,770 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,771 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5a0df8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:54,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:54,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,775 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:54,775 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:54,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:54,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:54,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:54,786 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:54,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:54,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:54,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:54,786 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:54,788 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,788 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,788 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:54,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,790 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,792 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:54,792 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:54,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:54,792 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,792 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:54,792 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:54,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:54,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:54,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:54,793 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:54,794 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:54,894 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:54,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:54,900 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:54,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:54,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:54,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:54,901 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,901 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,903 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:54,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:54,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,904 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,904 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:54,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:54,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:54,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:54,906 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,906 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:54,906 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:54,906 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:54,907 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:54,908 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f08ada[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:54,912 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:54,912 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:54,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:54,917 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:54,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:54,918 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:54,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:54,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:54,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:54,918 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:54,921 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,921 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:54,921 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:54,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,924 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:54,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:54,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:54,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:54,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:54,934 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:54,934 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:54,934 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:54,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:54,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:54,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219694963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219694964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:54,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:54,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219694964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,035 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:55,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:55,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:55,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:55,039 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:55,039 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,039 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:55,039 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:55,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,042 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,042 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:55,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:55,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:55,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,045 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:55,047 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,047 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:55,047 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:55,048 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:55,049 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:55,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54bb0d42[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:55,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:55,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,057 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,060 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,060 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:55,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:55,061 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:55,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:55,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:55,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:55,061 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:55,065 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,065 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,077 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,078 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,081 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:55,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,081 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:55,082 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:55,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,124 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/68b9791432364f68964a53a898966f76 2023-03-19 09:53:55,182 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:55,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:55,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:55,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:55,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:55,193 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:55,193 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,193 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:55,193 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:55,194 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,194 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:55,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:55,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:55,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:55,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:55,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:55,200 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:55,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,201 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64fdbdad[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:55,203 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:55,203 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,205 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:55,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:55,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:55,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:55,209 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:55,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:55,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:55,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:55,209 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:55,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:55,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:55,220 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,220 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,220 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,220 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,220 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:55,220 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:55,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:55,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219695266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219695268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219695269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,320 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:55,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:55,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:55,324 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:55,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:55,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:55,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,327 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:55,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:55,328 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,329 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:55,329 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:55,329 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:55,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:55,331 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f434907[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:55,333 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:55,333 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,335 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,335 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:55,335 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:55,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:55,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:55,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:55,340 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:55,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:55,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:55,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:55,340 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:55,342 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,342 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,342 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,349 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,349 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,349 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:55,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:55,349 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:55,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:55,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,349 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,349 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:55,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:55,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:55,453 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:55,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:55,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:55,454 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,454 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,463 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,463 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,463 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,463 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:55,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:55,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,467 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a00a86f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:55,469 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:55,469 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,471 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:55,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:55,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:55,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,473 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:55,476 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:55,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:55,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:55,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:55,476 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:55,477 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,477 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,477 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:55,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:55,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:55,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:55,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:55,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:55,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:55,483 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:55,483 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:55,483 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,483 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:55,483 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/8eb48457692446a48a4f20507614e714 2023-03-19 09:53:55,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/3b88cfa0384e42389821fa728f03b177 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3b88cfa0384e42389821fa728f03b177 2023-03-19 09:53:55,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3b88cfa0384e42389821fa728f03b177, entries=150, sequenceid=357, filesize=11.9 K 2023-03-19 09:53:55,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/68b9791432364f68964a53a898966f76 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/68b9791432364f68964a53a898966f76 2023-03-19 09:53:55,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/68b9791432364f68964a53a898966f76, entries=150, sequenceid=357, filesize=11.9 K 2023-03-19 09:53:55,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/8eb48457692446a48a4f20507614e714 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/8eb48457692446a48a4f20507614e714 2023-03-19 09:53:55,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/8eb48457692446a48a4f20507614e714, entries=150, sequenceid=357, filesize=11.9 K 2023-03-19 09:53:55,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 7ef6718d22690da9d245ca8892594cd1 in 947ms, sequenceid=357, compaction requested=true 2023-03-19 09:53:55,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:55,571 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:55,571 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:55,571 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:55,571 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:55,571 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:55,572 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:55,572 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:55,572 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,572 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:55,572 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b0e5394317d24e58878913b878bc2bff, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/eb07496d2d5945c589abecc962812408, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/68b9791432364f68964a53a898966f76] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=36.4 K 2023-03-19 09:53:55,572 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:55,572 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,572 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b3e21b0627747b88a52fa86e39cbca2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/5a7aa9ceda304597a364349b1b25ee67, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3b88cfa0384e42389821fa728f03b177] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=36.4 K 2023-03-19 09:53:55,573 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b0e5394317d24e58878913b878bc2bff, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679219632490 2023-03-19 09:53:55,573 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 4b3e21b0627747b88a52fa86e39cbca2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679219632490 2023-03-19 09:53:55,573 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting eb07496d2d5945c589abecc962812408, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1679219633146 2023-03-19 09:53:55,574 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5a7aa9ceda304597a364349b1b25ee67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1679219633146 2023-03-19 09:53:55,574 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 68b9791432364f68964a53a898966f76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679219634299 2023-03-19 09:53:55,574 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3b88cfa0384e42389821fa728f03b177, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679219634299 2023-03-19 09:53:55,582 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:55,583 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:55,583 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:55,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:55,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:55,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:55,593 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:55,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:55,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:55,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:55,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:55,597 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,597 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:55,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:55,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:55,598 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:55,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:55,599 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:53:55,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:55,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:55,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:55,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:55,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:55,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:55,613 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/1ad0c1da7fff4e969b2fdd6ba314d29d 2023-03-19 09:53:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:55,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:55,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219695788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219695789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219695789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219695890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219695892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:55,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:55,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219695893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,003 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/069bea8f256f4d1bb9d8de3b47e008b7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/069bea8f256f4d1bb9d8de3b47e008b7 2023-03-19 09:53:56,009 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/511ad60171d64efca65ec1252e9312a8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/511ad60171d64efca65ec1252e9312a8 2023-03-19 09:53:56,012 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into 069bea8f256f4d1bb9d8de3b47e008b7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:56,012 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:56,012 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=13, startTime=1679219635571; duration=0sec 2023-03-19 09:53:56,012 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:56,012 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:53:56,014 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:53:56,014 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:53:56,014 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:56,015 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/071d60ce5ebc4c74875c18545fce22af, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/ccf5803a8c444ec3b3cf8b0714e3f013, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/8eb48457692446a48a4f20507614e714] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=36.3 K 2023-03-19 09:53:56,015 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 071d60ce5ebc4c74875c18545fce22af, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679219632490 2023-03-19 09:53:56,015 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ccf5803a8c444ec3b3cf8b0714e3f013, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1679219633146 2023-03-19 09:53:56,016 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8eb48457692446a48a4f20507614e714, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679219634299 2023-03-19 09:53:56,020 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 511ad60171d64efca65ec1252e9312a8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:56,020 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:56,020 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=13, startTime=1679219635571; duration=0sec 2023-03-19 09:53:56,020 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:56,030 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/aa6b1f7c71814713871af1f7e7d3461c 2023-03-19 09:53:56,032 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#71 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:56,045 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/c850cd4dbe194818b75aa523c426306f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/c850cd4dbe194818b75aa523c426306f 2023-03-19 09:53:56,051 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into c850cd4dbe194818b75aa523c426306f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:56,051 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:56,051 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=13, startTime=1679219635571; duration=0sec 2023-03-19 09:53:56,051 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:56,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219696094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219696096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219696097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219696399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219696397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219696402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,446 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/cee6b6a5c0554356960351aa55e65ed2 2023-03-19 09:53:56,453 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/1ad0c1da7fff4e969b2fdd6ba314d29d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1ad0c1da7fff4e969b2fdd6ba314d29d 2023-03-19 09:53:56,461 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1ad0c1da7fff4e969b2fdd6ba314d29d, entries=150, sequenceid=372, filesize=11.9 K 2023-03-19 09:53:56,463 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/aa6b1f7c71814713871af1f7e7d3461c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa6b1f7c71814713871af1f7e7d3461c 2023-03-19 09:53:56,469 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa6b1f7c71814713871af1f7e7d3461c, entries=150, sequenceid=372, filesize=11.9 K 2023-03-19 09:53:56,470 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/cee6b6a5c0554356960351aa55e65ed2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/cee6b6a5c0554356960351aa55e65ed2 2023-03-19 09:53:56,480 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/cee6b6a5c0554356960351aa55e65ed2, entries=150, sequenceid=372, filesize=11.9 K 2023-03-19 09:53:56,481 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 7ef6718d22690da9d245ca8892594cd1 in 882ms, sequenceid=372, compaction requested=false 2023-03-19 09:53:56,481 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:56,481 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:56,484 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:56,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:56,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d6570[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:56,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:56,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:56,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:56,494 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:56,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:56,497 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:56,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:56,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:56,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:56,498 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:56,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:56,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:56,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:56,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:56,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:56,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:56,507 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:56,508 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:56,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:56,608 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:56,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:56,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:56,612 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:56,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:56,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:56,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:56,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:56,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:56,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:56,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:56,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:56,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:56,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:56,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:56,620 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:56,620 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:56,620 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:53:56,620 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:56,621 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:56,621 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:56,621 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:56,621 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:56,621 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:56,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:56,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:56,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219696923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219696924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:56,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:56,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219696924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219697027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219697028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219697028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,038 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/7caf952d8b29487494715ce2bd0c631d 2023-03-19 09:53:57,079 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/5d9830de6c994abb935fc884721c137d 2023-03-19 09:53:57,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219697229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219697232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219697234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,497 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/30a44c5621784425ac7f0f691788176b 2023-03-19 09:53:57,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/7caf952d8b29487494715ce2bd0c631d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7caf952d8b29487494715ce2bd0c631d 2023-03-19 09:53:57,508 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7caf952d8b29487494715ce2bd0c631d, entries=150, sequenceid=393, filesize=11.9 K 2023-03-19 09:53:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/5d9830de6c994abb935fc884721c137d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5d9830de6c994abb935fc884721c137d 2023-03-19 09:53:57,517 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5d9830de6c994abb935fc884721c137d, entries=150, sequenceid=393, filesize=11.9 K 2023-03-19 09:53:57,518 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/30a44c5621784425ac7f0f691788176b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/30a44c5621784425ac7f0f691788176b 2023-03-19 09:53:57,523 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/30a44c5621784425ac7f0f691788176b, entries=150, sequenceid=393, filesize=11.9 K 2023-03-19 09:53:57,524 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 7ef6718d22690da9d245ca8892594cd1 in 904ms, sequenceid=393, compaction requested=true 2023-03-19 09:53:57,525 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:57,525 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:57,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc4b0c3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:57,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:57,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:57,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,533 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:57,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:53:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:57,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:57,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:57,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:57,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:57,537 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:57,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:57,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:57,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:57,538 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:57,542 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,542 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,542 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,549 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:57,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,551 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:57,552 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:57,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:57,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219697580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219697583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219697586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,652 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:57,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:57,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:57,656 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:57,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:57,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:57,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:57,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,658 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:57,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:57,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:57,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:57,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:57,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:57,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:57,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:57,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:57,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:57,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:57,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:57,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3acb4d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:57,667 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:57,667 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,672 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:57,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,673 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:57,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:57,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:57,677 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:57,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:57,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:57,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:57,677 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:57,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,684 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:57,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:57,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:57,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:57,684 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:57,684 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:57,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219697686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219697687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219697687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,784 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:57,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:57,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:57,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:57,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:57,788 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:57,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:57,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:57,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:57,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,790 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,790 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:57,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:57,790 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,790 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:57,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:57,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:57,792 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:57,792 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:57,792 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:57,792 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:57,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:57,794 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,796 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,796 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:57,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757647b5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:57,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:57,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,800 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:57,800 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,800 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:57,803 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:57,806 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:57,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:57,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:57,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:57,806 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:57,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,814 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:57,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:57,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,815 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,815 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:57,815 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:57,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219697889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219697889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:57,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219697889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,915 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:57,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:57,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:57,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:57,919 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:57,919 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:57,919 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:57,919 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:57,919 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,919 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,921 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:57,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:57,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:57,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:57,923 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:57,923 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:57,923 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:57,923 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:57,923 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:57,923 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:57,923 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,923 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:57,923 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:57,925 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186dca8e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:57,927 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:57,927 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:57,929 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,929 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:57,930 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,930 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,930 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,930 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:57,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:57,933 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:57,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:57,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:57,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:57,933 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:57,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:57,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:57,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:57,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:57,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:57,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:57,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:57,944 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:57,944 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:57,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:57,962 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/4030da20060a4d609a7c5090914c71cb 2023-03-19 09:53:57,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/616fcebc9eca4b2cb9f2d6b38bcacb29 2023-03-19 09:53:57,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/2ee1dc61458f4ff78a522959eec2bf27 2023-03-19 09:53:57,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/4030da20060a4d609a7c5090914c71cb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4030da20060a4d609a7c5090914c71cb 2023-03-19 09:53:58,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4030da20060a4d609a7c5090914c71cb, entries=200, sequenceid=409, filesize=14.3 K 2023-03-19 09:53:58,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/616fcebc9eca4b2cb9f2d6b38bcacb29 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/616fcebc9eca4b2cb9f2d6b38bcacb29 2023-03-19 09:53:58,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/616fcebc9eca4b2cb9f2d6b38bcacb29, entries=150, sequenceid=409, filesize=11.9 K 2023-03-19 09:53:58,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/2ee1dc61458f4ff78a522959eec2bf27 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/2ee1dc61458f4ff78a522959eec2bf27 2023-03-19 09:53:58,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/2ee1dc61458f4ff78a522959eec2bf27, entries=150, sequenceid=409, filesize=11.9 K 2023-03-19 09:53:58,015 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 7ef6718d22690da9d245ca8892594cd1 in 481ms, sequenceid=409, compaction requested=true 2023-03-19 09:53:58,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:58,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:58,015 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:58,015 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:58,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:58,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:58,017 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:58,017 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:58,017 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:58,017 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/511ad60171d64efca65ec1252e9312a8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1ad0c1da7fff4e969b2fdd6ba314d29d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7caf952d8b29487494715ce2bd0c631d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4030da20060a4d609a7c5090914c71cb] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=50.8 K 2023-03-19 09:53:58,018 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 511ad60171d64efca65ec1252e9312a8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679219634299 2023-03-19 09:53:58,018 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:58,018 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:58,018 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1ad0c1da7fff4e969b2fdd6ba314d29d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679219634638 2023-03-19 09:53:58,018 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:58,018 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/069bea8f256f4d1bb9d8de3b47e008b7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa6b1f7c71814713871af1f7e7d3461c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5d9830de6c994abb935fc884721c137d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/616fcebc9eca4b2cb9f2d6b38bcacb29] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=48.4 K 2023-03-19 09:53:58,018 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7caf952d8b29487494715ce2bd0c631d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1679219635784 2023-03-19 09:53:58,019 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 069bea8f256f4d1bb9d8de3b47e008b7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679219634299 2023-03-19 09:53:58,019 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 4030da20060a4d609a7c5090914c71cb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679219636917 2023-03-19 09:53:58,019 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting aa6b1f7c71814713871af1f7e7d3461c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679219634638 2023-03-19 09:53:58,020 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5d9830de6c994abb935fc884721c137d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1679219635784 2023-03-19 09:53:58,020 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 616fcebc9eca4b2cb9f2d6b38bcacb29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679219636921 2023-03-19 09:53:58,028 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:58,035 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#80 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:58,041 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/3eadccb8a4844df485cfcd4088d05911 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3eadccb8a4844df485cfcd4088d05911 2023-03-19 09:53:58,044 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:58,047 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/1e827bc0451c4e2a870411b7026af716 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/1e827bc0451c4e2a870411b7026af716 2023-03-19 09:53:58,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:58,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:58,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:58,050 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:58,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:58,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:58,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:58,051 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:58,051 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:58,052 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 3eadccb8a4844df485cfcd4088d05911(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:58,052 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:58,052 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=12, startTime=1679219638015; duration=0sec 2023-03-19 09:53:58,052 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:58,052 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:58,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,053 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:58,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:58,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:58,054 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:58,054 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:53:58,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:58,054 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:58,054 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/c850cd4dbe194818b75aa523c426306f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/cee6b6a5c0554356960351aa55e65ed2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/30a44c5621784425ac7f0f691788176b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/2ee1dc61458f4ff78a522959eec2bf27] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=48.3 K 2023-03-19 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:58,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:58,055 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c850cd4dbe194818b75aa523c426306f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679219634299 2023-03-19 09:53:58,055 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting cee6b6a5c0554356960351aa55e65ed2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679219634638 2023-03-19 09:53:58,056 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 30a44c5621784425ac7f0f691788176b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1679219635784 2023-03-19 09:53:58,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:58,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:58,056 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2ee1dc61458f4ff78a522959eec2bf27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679219636921 2023-03-19 09:53:58,057 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into 1e827bc0451c4e2a870411b7026af716(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:58,057 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:58,058 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=12, startTime=1679219638015; duration=0sec 2023-03-19 09:53:58,058 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:58,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:58,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:58,066 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:58,067 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:58,067 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:53:58,067 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:58,067 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:58,067 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:58,067 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:58,067 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:58,067 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:58,068 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:58,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:58,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:58,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219698236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219698237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219698238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219698339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219698343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219698347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,497 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/a02b8982d1c34beeaab58b23198e2c5e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a02b8982d1c34beeaab58b23198e2c5e 2023-03-19 09:53:58,501 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/9c6bb985f85c4321abd6524697dd8209 2023-03-19 09:53:58,511 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into a02b8982d1c34beeaab58b23198e2c5e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:53:58,511 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:58,511 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=12, startTime=1679219638015; duration=0sec 2023-03-19 09:53:58,511 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:58,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219698544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219698547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219698550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219698849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219698850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219698856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,919 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/de46489d819d4925bf292bd522199492 2023-03-19 09:53:58,944 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/b02234d6d6e444e2b61278b8c4739ea8 2023-03-19 09:53:58,951 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/9c6bb985f85c4321abd6524697dd8209 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9c6bb985f85c4321abd6524697dd8209 2023-03-19 09:53:58,958 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9c6bb985f85c4321abd6524697dd8209, entries=150, sequenceid=427, filesize=11.9 K 2023-03-19 09:53:58,960 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/de46489d819d4925bf292bd522199492 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/de46489d819d4925bf292bd522199492 2023-03-19 09:53:58,970 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/de46489d819d4925bf292bd522199492, entries=150, sequenceid=427, filesize=11.9 K 2023-03-19 09:53:58,971 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/b02234d6d6e444e2b61278b8c4739ea8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/b02234d6d6e444e2b61278b8c4739ea8 2023-03-19 09:53:58,982 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/b02234d6d6e444e2b61278b8c4739ea8, entries=150, sequenceid=427, filesize=11.9 K 2023-03-19 09:53:58,983 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 7ef6718d22690da9d245ca8892594cd1 in 916ms, sequenceid=427, compaction requested=false 2023-03-19 09:53:58,983 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:58,983 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:58,983 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:58,983 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:58,983 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:58,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:58,986 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:58,986 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:58,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:58,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@320533e8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:58,990 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:58,990 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:58,992 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:58,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:58,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:58,998 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:58,999 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:58,999 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:59,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:59,004 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:59,004 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:59,004 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:59,004 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:59,004 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:59,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,017 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,020 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:59,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,020 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,020 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:59,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,020 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,020 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:59,020 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:59,020 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,020 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,120 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:59,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:59,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:59,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:59,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:59,124 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:59,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:59,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:59,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,126 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:59,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:59,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:59,128 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,128 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:59,129 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:53:59,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:59,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:59,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:59,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:59,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:59,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:59,146 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/bd0b39e8d61649959f728c23c66a8fcd 2023-03-19 09:53:59,161 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/b55ae9832543470394bd50801fd1003e 2023-03-19 09:53:59,181 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/a7bb78c9abee4403b235c1880937688e 2023-03-19 09:53:59,201 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/bd0b39e8d61649959f728c23c66a8fcd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/bd0b39e8d61649959f728c23c66a8fcd 2023-03-19 09:53:59,209 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/bd0b39e8d61649959f728c23c66a8fcd, entries=150, sequenceid=444, filesize=11.9 K 2023-03-19 09:53:59,210 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/b55ae9832543470394bd50801fd1003e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b55ae9832543470394bd50801fd1003e 2023-03-19 09:53:59,216 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b55ae9832543470394bd50801fd1003e, entries=150, sequenceid=444, filesize=11.9 K 2023-03-19 09:53:59,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/a7bb78c9abee4403b235c1880937688e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a7bb78c9abee4403b235c1880937688e 2023-03-19 09:53:59,226 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a7bb78c9abee4403b235c1880937688e, entries=150, sequenceid=444, filesize=11.9 K 2023-03-19 09:53:59,228 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 7ef6718d22690da9d245ca8892594cd1 in 100ms, sequenceid=444, compaction requested=true 2023-03-19 09:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:59,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,235 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:59,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e49e646[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:59,239 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:59,239 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:59,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:59,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:59,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:59,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:59,244 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:59,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:59,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:59,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:59,244 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:59,246 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,246 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,247 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,247 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,253 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,253 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:59,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:59,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,254 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:59,254 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:59,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,355 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:59,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:59,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:59,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:53:59,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:53:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:53:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:53:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:53:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:53:59,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:59,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:59,372 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:59,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:59,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:59,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:59,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:59,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:59,375 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:59,375 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:59,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:59,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:59,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:59,380 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594551d5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:59,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:59,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:59,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:59,387 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:59,390 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:59,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:59,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:59,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:59,390 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:59,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:59,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:59,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:59,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:59,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219699410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219699412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219699412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/ee4e9b9478c545ea8bf003d6e4b92402 2023-03-19 09:53:59,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/f6b4f1d2b0824247a46c41168417b5ea 2023-03-19 09:53:59,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:59,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:59,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:59,502 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:59,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:59,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:59,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:59,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:59,507 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,507 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:59,507 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:59,507 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:59,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,511 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,511 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8ac9c9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:59,512 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:59,512 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,514 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:59,514 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,514 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:59,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219699513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,515 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219699518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219699519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:59,520 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:59,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:59,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:59,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:59,520 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:59,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,527 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:59,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:59,539 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:59,539 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:59,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,639 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:59,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:59,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:59,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:59,642 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:59,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:59,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:59,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,644 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,645 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,645 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:59,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:59,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:59,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:59,647 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,647 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:59,647 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:59,647 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,648 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,649 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,649 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:59,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@348b25a3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:59,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:59,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,651 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:59,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:59,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:59,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:59,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:59,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:59,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:59,656 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:59,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:59,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:59,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:59,656 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:59,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,662 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:59,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:59,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:59,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,662 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:59,662 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:59,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219699716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219699720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:53:59,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219699720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,762 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:59,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:59,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:59,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:59,766 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:59,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:59,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:59,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,768 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:59,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:59,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:59,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:59,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:59,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:59,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65219ea2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:59,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:59,776 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:59,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:59,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:59,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:59,779 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:59,781 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:59,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:59,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:59,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:59,781 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:59,783 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,783 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,783 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:59,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:59,791 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,792 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:59,792 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:59,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,792 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,796 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,796 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/90e0b4d83fa44e57be42d8904e647444 2023-03-19 09:53:59,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/ee4e9b9478c545ea8bf003d6e4b92402 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee4e9b9478c545ea8bf003d6e4b92402 2023-03-19 09:53:59,892 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:53:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:53:59,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee4e9b9478c545ea8bf003d6e4b92402, entries=350, sequenceid=456, filesize=21.4 K 2023-03-19 09:53:59,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:53:59,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/f6b4f1d2b0824247a46c41168417b5ea as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f6b4f1d2b0824247a46c41168417b5ea 2023-03-19 09:53:59,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:53:59,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:53:59,895 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:53:59,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:53:59,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:53:59,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,901 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,901 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,901 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:53:59,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:53:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:53:59,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f6b4f1d2b0824247a46c41168417b5ea, entries=150, sequenceid=456, filesize=11.9 K 2023-03-19 09:53:59,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:53:59,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:53:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:53:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:53:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:53:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:53:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:53:59,903 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,903 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:53:59,903 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:53:59,903 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:53:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:53:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:53:59,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/90e0b4d83fa44e57be42d8904e647444 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/90e0b4d83fa44e57be42d8904e647444 2023-03-19 09:53:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:53:59,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:53:59,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1619a02a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:53:59,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:53:59,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/90e0b4d83fa44e57be42d8904e647444, entries=150, sequenceid=456, filesize=11.9 K 2023-03-19 09:53:59,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:53:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:53:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:53:59,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:53:59,910 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7ef6718d22690da9d245ca8892594cd1 in 544ms, sequenceid=456, compaction requested=true 2023-03-19 09:53:59,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:53:59,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:59,910 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:59,910 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:53:59,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:53:59,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:53:59,911 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59428 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:59,911 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:53:59,911 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:53:59,911 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:53:59,912 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,912 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:53:59,912 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3eadccb8a4844df485cfcd4088d05911, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9c6bb985f85c4321abd6524697dd8209, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/bd0b39e8d61649959f728c23c66a8fcd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee4e9b9478c545ea8bf003d6e4b92402] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=58.0 K 2023-03-19 09:53:59,912 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/1e827bc0451c4e2a870411b7026af716, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/de46489d819d4925bf292bd522199492, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b55ae9832543470394bd50801fd1003e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f6b4f1d2b0824247a46c41168417b5ea] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=48.5 K 2023-03-19 09:53:59,912 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3eadccb8a4844df485cfcd4088d05911, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679219636921 2023-03-19 09:53:59,912 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1e827bc0451c4e2a870411b7026af716, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679219636921 2023-03-19 09:53:59,913 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9c6bb985f85c4321abd6524697dd8209, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679219637578 2023-03-19 09:53:59,913 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting de46489d819d4925bf292bd522199492, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679219637578 2023-03-19 09:53:59,913 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,913 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting bd0b39e8d61649959f728c23c66a8fcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1679219638204 2023-03-19 09:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:53:59,913 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b55ae9832543470394bd50801fd1003e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1679219638204 2023-03-19 09:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:53:59,914 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ee4e9b9478c545ea8bf003d6e4b92402, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679219639357 2023-03-19 09:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,914 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f6b4f1d2b0824247a46c41168417b5ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679219639362 2023-03-19 09:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:53:59,917 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:53:59,917 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:53:59,917 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:53:59,917 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:53:59,917 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:53:59,918 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,918 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:53:59,918 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:53:59,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,919 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:53:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,925 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:53:59,925 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:53:59,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:53:59,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:53:59,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:53:59,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:53:59,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:53:59,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:53:59,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:53:59,926 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:53:59,927 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:53:59,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:53:59,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:53:59,928 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:53:59,928 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:53:59,928 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#91 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:00,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:00,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-19 09:54:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:54:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:54:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:54:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:00,028 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:00,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:00,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:00,032 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:00,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:00,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:00,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,036 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,036 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:00,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/cb25c55f14f947259444c237c2d4561a 2023-03-19 09:54:00,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:00,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:00,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:00,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:00,039 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa4f7a1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:00,042 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:00,042 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:00,044 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:00,044 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:00,044 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:00,046 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:00,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:00,049 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:00,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:00,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:00,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:00,050 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:00,053 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,053 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,053 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219700047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219700048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,055 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219700053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,058 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:00,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:00,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,058 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:00,058 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:00,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/aa04e1534bb84f2c908003135a4eaff1 2023-03-19 09:54:00,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219700154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219700156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219700156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,159 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:00,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:00,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:00,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:00,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:00,162 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:00,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:00,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:00,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,164 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,164 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,164 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:00,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:00,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,165 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:00,166 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,166 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:00,166 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:00,166 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:00,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,167 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:00,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef71cb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:00,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:00,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:00,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:00,174 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:00,177 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:00,177 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:00,177 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:00,177 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:00,177 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:00,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,181 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,183 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:00,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:00,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:00,184 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:00,184 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,184 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,184 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,184 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,284 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:00,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:00,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:00,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:00,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:00,287 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:00,287 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,287 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:00,287 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:00,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:00,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:00,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:00,292 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,292 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:00,292 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:00,292 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:00,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:00,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:00,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b522bba[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:00,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:00,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:00,302 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:00,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:00,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:00,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:00,302 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:00,305 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,305 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,305 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,306 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,307 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:00,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:00,309 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,310 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:00,310 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:00,346 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/cc35d17803d84853a900069c472aabe0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/cc35d17803d84853a900069c472aabe0 2023-03-19 09:54:00,346 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/93dc8461742f4288850df7e66adbd065 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/93dc8461742f4288850df7e66adbd065 2023-03-19 09:54:00,353 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 93dc8461742f4288850df7e66adbd065(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:00,354 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:00,354 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=12, startTime=1679219639910; duration=0sec 2023-03-19 09:54:00,354 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into cc35d17803d84853a900069c472aabe0(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:00,354 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:00,354 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:00,354 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=12, startTime=1679219639910; duration=0sec 2023-03-19 09:54:00,354 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:00,354 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:00,355 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:00,355 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:54:00,355 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,355 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a02b8982d1c34beeaab58b23198e2c5e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/b02234d6d6e444e2b61278b8c4739ea8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a7bb78c9abee4403b235c1880937688e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/90e0b4d83fa44e57be42d8904e647444] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=48.4 K 2023-03-19 09:54:00,356 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a02b8982d1c34beeaab58b23198e2c5e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679219636921 2023-03-19 09:54:00,356 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b02234d6d6e444e2b61278b8c4739ea8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679219637578 2023-03-19 09:54:00,356 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a7bb78c9abee4403b235c1880937688e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1679219638204 2023-03-19 09:54:00,357 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 90e0b4d83fa44e57be42d8904e647444, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679219639362 2023-03-19 09:54:00,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219700359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219700359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219700360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,371 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:00,410 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:00,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:00,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:00,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:00,418 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:00,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:00,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:00,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,423 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,423 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:00,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:00,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:00,428 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:00,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3011ce0a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:00,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:00,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:00,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:00,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:00,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:00,443 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:00,443 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:00,443 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:00,443 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:00,444 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:00,446 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,446 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:00,446 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:00,447 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,452 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:00,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:00,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:00,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:00,452 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:00,452 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:00,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:00,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:00,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:00,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/0c28475f2a5f455b9cc82f7185645255 2023-03-19 09:54:00,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/cb25c55f14f947259444c237c2d4561a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/cb25c55f14f947259444c237c2d4561a 2023-03-19 09:54:00,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/cb25c55f14f947259444c237c2d4561a, entries=150, sequenceid=477, filesize=11.9 K 2023-03-19 09:54:00,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/aa04e1534bb84f2c908003135a4eaff1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa04e1534bb84f2c908003135a4eaff1 2023-03-19 09:54:00,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa04e1534bb84f2c908003135a4eaff1, entries=150, sequenceid=477, filesize=11.9 K 2023-03-19 09:54:00,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/0c28475f2a5f455b9cc82f7185645255 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0c28475f2a5f455b9cc82f7185645255 2023-03-19 09:54:00,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0c28475f2a5f455b9cc82f7185645255, entries=150, sequenceid=477, filesize=11.9 K 2023-03-19 09:54:00,514 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 7ef6718d22690da9d245ca8892594cd1 in 489ms, sequenceid=477, compaction requested=false 2023-03-19 09:54:00,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:00,552 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:00,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:00,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:00,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:00,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:00,556 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:00,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:00,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:00,557 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,557 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,558 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,558 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:00,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:00,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:00,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:00,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:00,560 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:00,560 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:00,560 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-19 09:54:00,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:54:00,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:00,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:54:00,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:00,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:54:00,561 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:00,572 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/68bbc3d2dd384093bf7202d64bfdb853 2023-03-19 09:54:00,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:00,666 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:00,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219700689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219700689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219700690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,789 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/32edc451b08b4cee81d7419361a61257 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/32edc451b08b4cee81d7419361a61257 2023-03-19 09:54:00,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219700796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219700796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219700796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:00,797 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into 32edc451b08b4cee81d7419361a61257(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:00,797 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:00,797 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=12, startTime=1679219639910; duration=0sec 2023-03-19 09:54:00,798 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:00,993 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/07d926b9b6244fe4b3af06d76d45968b 2023-03-19 09:54:00,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219700999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219700999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219701004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,015 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/23d3956a4b2f40968768f28270f3f29e 2023-03-19 09:54:01,020 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/68bbc3d2dd384093bf7202d64bfdb853 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/68bbc3d2dd384093bf7202d64bfdb853 2023-03-19 09:54:01,025 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/68bbc3d2dd384093bf7202d64bfdb853, entries=150, sequenceid=491, filesize=11.9 K 2023-03-19 09:54:01,025 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/07d926b9b6244fe4b3af06d76d45968b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/07d926b9b6244fe4b3af06d76d45968b 2023-03-19 09:54:01,030 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/07d926b9b6244fe4b3af06d76d45968b, entries=150, sequenceid=491, filesize=11.9 K 2023-03-19 09:54:01,031 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/23d3956a4b2f40968768f28270f3f29e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/23d3956a4b2f40968768f28270f3f29e 2023-03-19 09:54:01,036 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/23d3956a4b2f40968768f28270f3f29e, entries=150, sequenceid=491, filesize=11.9 K 2023-03-19 09:54:01,037 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7ef6718d22690da9d245ca8892594cd1 in 477ms, sequenceid=491, compaction requested=true 2023-03-19 09:54:01,037 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:01,037 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:01,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:01,039 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,039 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c02e56[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:01,041 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:01,041 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,042 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:01,042 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:01,042 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,042 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:01,044 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,044 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:01,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:01,047 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:01,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:01,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:01,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:01,047 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:01,049 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:01,049 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,050 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,050 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:01,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:01,052 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,054 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:01,054 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:01,054 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:01,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:01,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:01,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:01,054 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:01,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:01,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:01,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,055 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:01,055 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:01,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:01,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:01,155 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:01,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:01,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:01,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:01,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:01,159 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:01,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:01,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:01,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:01,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,160 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:01,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:01,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:01,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:01,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:01,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:01,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:01,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:01,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:01,162 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:01,162 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:01,162 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:01,163 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:01,163 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:01,163 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:01,163 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:01,163 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-19 09:54:01,163 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:54:01,163 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:01,163 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:54:01,163 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:01,164 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:54:01,164 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:01,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219701312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219701313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219701315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219701418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219701418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219701418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,572 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/61b9c4714dfa4fae841b0cb42905ba50 2023-03-19 09:54:01,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219701621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219701621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219701622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219701923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219701925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219701925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:01,984 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/770df38fee1b4ede9b189b6956f48971 2023-03-19 09:54:02,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32958 deadline: 1679219702151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,154 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18212 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:54:02,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32916 deadline: 1679219702183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,187 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18239 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:54:02,400 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/12ce4e06ce474517a41c77e585caf1bd 2023-03-19 09:54:02,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/61b9c4714dfa4fae841b0cb42905ba50 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/61b9c4714dfa4fae841b0cb42905ba50 2023-03-19 09:54:02,417 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/61b9c4714dfa4fae841b0cb42905ba50, entries=150, sequenceid=512, filesize=11.9 K 2023-03-19 09:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/770df38fee1b4ede9b189b6956f48971 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/770df38fee1b4ede9b189b6956f48971 2023-03-19 09:54:02,424 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/770df38fee1b4ede9b189b6956f48971, entries=150, sequenceid=512, filesize=11.9 K 2023-03-19 09:54:02,425 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/12ce4e06ce474517a41c77e585caf1bd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/12ce4e06ce474517a41c77e585caf1bd 2023-03-19 09:54:02,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32932 deadline: 1679219702426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,429 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/12ce4e06ce474517a41c77e585caf1bd, entries=150, sequenceid=512, filesize=11.9 K 2023-03-19 09:54:02,431 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7ef6718d22690da9d245ca8892594cd1 in 1268ms, sequenceid=512, compaction requested=true 2023-03-19 09:54:02,431 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:02,431 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:02,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:02,434 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,434 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d879ac[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:02,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:02,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,441 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,441 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:02,441 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:02,441 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:02,442 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:02,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:02,448 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:02,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:02,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:02,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:02,448 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:02,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,451 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:02,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:02,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:02,461 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,461 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:02,461 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:02,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:02,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219702467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219702470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,562 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:02,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:02,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:02,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:02,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:02,566 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:02,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:02,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:02,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:02,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:02,567 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:02,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:02,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:02,571 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b2d9dd6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:02,574 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:02,574 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219702572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219702575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,580 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:02,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,581 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:02,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:02,584 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:02,584 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:02,584 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:02,584 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:02,584 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:02,586 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,586 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,586 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,586 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,588 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:02,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:02,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:02,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,591 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:02,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:02,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,591 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,691 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:02,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:02,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:02,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:02,700 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:02,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:02,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:02,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:02,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,703 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:02,704 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:02,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:02,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:02,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:02,714 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:02,714 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:02,714 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:02,714 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:02,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f26d27[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:02,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:02,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,730 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:02,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:02,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:02,731 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,731 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:02,733 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:02,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:02,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:02,737 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:02,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:02,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:02,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:02,737 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:02,740 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,740 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,740 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:02,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:02,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:02,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,749 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:02,749 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:02,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219702777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:02,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219702778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,847 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/2df05bff1ed6480faa069a52524d8080 2023-03-19 09:54:02,849 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:02,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:02,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:02,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:02,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:02,858 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:02,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:02,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:02,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:02,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:02,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:02,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:02,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:02,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:02,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:02,865 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:02,867 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,868 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:02,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb95951[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:02,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:02,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:02,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:02,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:02,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:02,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:02,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:02,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:02,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:02,885 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:02,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:02,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:02,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:02,886 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:02,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:02,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,892 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:02,896 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:02,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:02,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:02,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:02,897 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:02,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:02,897 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:02,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:02,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:02,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:02,997 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:02,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:03,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:03,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:03,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:03,002 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:03,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:03,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:03,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:03,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:03,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:03,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:03,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:03,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:03,007 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:03,008 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,008 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:03,008 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:03,008 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:03,010 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eee5273[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:03,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:03,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:03,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:03,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:03,015 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,015 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:03,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:03,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:03,023 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:03,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:03,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:03,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:03,023 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:03,027 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,027 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,027 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:03,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,029 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:03,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:03,038 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:03,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,038 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:03,038 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:03,039 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:03,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:03,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:03,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:03,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:03,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32888 deadline: 1679219703084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:03,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32872 deadline: 1679219703084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:03,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:03,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:03,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:03,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:03,142 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:03,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:03,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:03,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,144 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:03,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:03,145 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,145 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,145 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:03,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:03,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:03,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:03,150 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:03,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:03,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@693ab355[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:03,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,153 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:03,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:03,155 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:03,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:03,158 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:03,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:03,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:03,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:03,158 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:03,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:03,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:03,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:03,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,168 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:03,168 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:03,167 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:03,172 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:03,172 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:03,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:03,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:03,268 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:03,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:03,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/f9ef7089d10d4895832d3cb26d881227 2023-03-19 09:54:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:03,271 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:03,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:03,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:03,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,273 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:03,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:03,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:03,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:03,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:03,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:03,276 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,276 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:03,276 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:03,276 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:03,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:03,278 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:03,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff77000[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:03,282 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:03,282 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:03,283 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,283 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:03,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:03,290 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:03,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:03,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:03,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:03,290 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:03,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:03,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:03,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,294 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/96775c4d4cf1469e887a4af50a179999 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:03,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:03,299 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:03,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,299 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:03,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/2df05bff1ed6480faa069a52524d8080 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/2df05bff1ed6480faa069a52524d8080 2023-03-19 09:54:03,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/2df05bff1ed6480faa069a52524d8080, entries=250, sequenceid=526, filesize=16.7 K 2023-03-19 09:54:03,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/f9ef7089d10d4895832d3cb26d881227 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f9ef7089d10d4895832d3cb26d881227 2023-03-19 09:54:03,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f9ef7089d10d4895832d3cb26d881227, entries=150, sequenceid=526, filesize=11.9 K 2023-03-19 09:54:03,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/96775c4d4cf1469e887a4af50a179999 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/96775c4d4cf1469e887a4af50a179999 2023-03-19 09:54:03,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/96775c4d4cf1469e887a4af50a179999, entries=150, sequenceid=526, filesize=11.9 K 2023-03-19 09:54:03,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7ef6718d22690da9d245ca8892594cd1 in 889ms, sequenceid=526, compaction requested=true 2023-03-19 09:54:03,323 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:03,323 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:03,323 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:54:03,323 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:54:03,323 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:03,323 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:03,326 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66866 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:54:03,326 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:54:03,326 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/A is initiating minor compaction (all files) 2023-03-19 09:54:03,326 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/B is initiating minor compaction (all files) 2023-03-19 09:54:03,326 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/A in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,326 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/B in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,326 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/93dc8461742f4288850df7e66adbd065, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/cb25c55f14f947259444c237c2d4561a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/68bbc3d2dd384093bf7202d64bfdb853, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/61b9c4714dfa4fae841b0cb42905ba50, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/2df05bff1ed6480faa069a52524d8080] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=65.3 K 2023-03-19 09:54:03,326 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/cc35d17803d84853a900069c472aabe0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa04e1534bb84f2c908003135a4eaff1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/07d926b9b6244fe4b3af06d76d45968b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/770df38fee1b4ede9b189b6956f48971, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f9ef7089d10d4895832d3cb26d881227] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=60.5 K 2023-03-19 09:54:03,327 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 93dc8461742f4288850df7e66adbd065, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679219639362 2023-03-19 09:54:03,327 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting cc35d17803d84853a900069c472aabe0, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679219639362 2023-03-19 09:54:03,328 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting cb25c55f14f947259444c237c2d4561a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219639402 2023-03-19 09:54:03,328 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting aa04e1534bb84f2c908003135a4eaff1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219639402 2023-03-19 09:54:03,329 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 68bbc3d2dd384093bf7202d64bfdb853, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1679219640037 2023-03-19 09:54:03,329 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 07d926b9b6244fe4b3af06d76d45968b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1679219640037 2023-03-19 09:54:03,329 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 61b9c4714dfa4fae841b0cb42905ba50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679219640682 2023-03-19 09:54:03,330 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 770df38fee1b4ede9b189b6956f48971, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679219640682 2023-03-19 09:54:03,330 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2df05bff1ed6480faa069a52524d8080, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679219641309 2023-03-19 09:54:03,332 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f9ef7089d10d4895832d3cb26d881227, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679219641314 2023-03-19 09:54:03,353 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#A#compaction#106 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:03,362 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#B#compaction#107 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:03,399 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:03,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:03,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:03,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:03,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:03,402 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:03,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,403 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:03,403 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:03,403 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,403 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:03,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:03,405 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:03,405 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:03,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:03,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:03,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. started... 2023-03-19 09:54:03,406 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:54:03,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:54:03,407 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:03,407 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:54:03,407 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:03,407 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:54:03,407 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:03,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:03,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. as already flushing 2023-03-19 09:54:03,446 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:54245 2023-03-19 09:54:03,446 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:03,446 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:54245 2023-03-19 09:54:03,446 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:03,448 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:54245 2023-03-19 09:54:03,449 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:03,449 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:54245 2023-03-19 09:54:03,449 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:03,453 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:54245 2023-03-19 09:54:03,453 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:03,590 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:54245 2023-03-19 09:54:03,590 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:54245 2023-03-19 09:54:03,591 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:03,591 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:03,771 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/1bb2657de2e94f689203361df7747974 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1bb2657de2e94f689203361df7747974 2023-03-19 09:54:03,776 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/A of 7ef6718d22690da9d245ca8892594cd1 into 1bb2657de2e94f689203361df7747974(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:03,776 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:03,776 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/A, priority=11, startTime=1679219643323; duration=0sec 2023-03-19 09:54:03,776 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:03,776 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:54:03,777 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:54:03,778 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 7ef6718d22690da9d245ca8892594cd1/C is initiating minor compaction (all files) 2023-03-19 09:54:03,778 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7ef6718d22690da9d245ca8892594cd1/C in TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:03,778 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/32edc451b08b4cee81d7419361a61257, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0c28475f2a5f455b9cc82f7185645255, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/23d3956a4b2f40968768f28270f3f29e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/12ce4e06ce474517a41c77e585caf1bd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/96775c4d4cf1469e887a4af50a179999] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp, totalSize=60.5 K 2023-03-19 09:54:03,778 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 32edc451b08b4cee81d7419361a61257, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679219639362 2023-03-19 09:54:03,778 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0c28475f2a5f455b9cc82f7185645255, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219639402 2023-03-19 09:54:03,779 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 23d3956a4b2f40968768f28270f3f29e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1679219640037 2023-03-19 09:54:03,779 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 12ce4e06ce474517a41c77e585caf1bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679219640682 2023-03-19 09:54:03,780 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 96775c4d4cf1469e887a4af50a179999, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679219641314 2023-03-19 09:54:03,783 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/8465f12688aa4c98a6360ff3da39b67d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/8465f12688aa4c98a6360ff3da39b67d 2023-03-19 09:54:03,788 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/B of 7ef6718d22690da9d245ca8892594cd1 into 8465f12688aa4c98a6360ff3da39b67d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:03,788 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:03,788 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/B, priority=11, startTime=1679219643323; duration=0sec 2023-03-19 09:54:03,788 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:03,789 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7ef6718d22690da9d245ca8892594cd1#C#compaction#109 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:03,798 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/80dbeca409a943588af85bbfd69a998f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/80dbeca409a943588af85bbfd69a998f 2023-03-19 09:54:03,802 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7ef6718d22690da9d245ca8892594cd1/C of 7ef6718d22690da9d245ca8892594cd1 into 80dbeca409a943588af85bbfd69a998f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:03,802 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:03,802 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1., storeName=7ef6718d22690da9d245ca8892594cd1/C, priority=11, startTime=1679219643323; duration=0sec 2023-03-19 09:54:03,802 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:03,818 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/c71ef7768bdc48798bf9463e5a8ac338 2023-03-19 09:54:04,230 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/8d5c0d5eaf3d46c883a922c4155994da 2023-03-19 09:54:04,647 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/aa120617d146424b8ef7af7c4115fe66 2023-03-19 09:54:04,653 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/c71ef7768bdc48798bf9463e5a8ac338 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c71ef7768bdc48798bf9463e5a8ac338 2023-03-19 09:54:04,658 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c71ef7768bdc48798bf9463e5a8ac338, entries=150, sequenceid=544, filesize=11.9 K 2023-03-19 09:54:04,659 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/8d5c0d5eaf3d46c883a922c4155994da as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/8d5c0d5eaf3d46c883a922c4155994da 2023-03-19 09:54:04,665 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/8d5c0d5eaf3d46c883a922c4155994da, entries=150, sequenceid=544, filesize=11.9 K 2023-03-19 09:54:04,666 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/aa120617d146424b8ef7af7c4115fe66 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/aa120617d146424b8ef7af7c4115fe66 2023-03-19 09:54:04,671 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/aa120617d146424b8ef7af7c4115fe66, entries=150, sequenceid=544, filesize=11.9 K 2023-03-19 09:54:04,672 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=33.54 KB/34350 for 7ef6718d22690da9d245ca8892594cd1 in 1266ms, sequenceid=544, compaction requested=false 2023-03-19 09:54:04,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:04,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:04,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:04,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:04,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:04,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:04,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:04,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:04,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:04,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:04,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4972d57e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:04,676 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:04,676 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,677 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:04,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:04,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:04,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:04,683 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:04,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:04,686 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:04,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:04,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:04,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:04,686 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:04,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:04,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:04,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:04,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:04,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:04,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:04,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:04,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:04,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:04,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:04,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:04,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:04,694 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:04,694 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:04,795 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:04,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:12,214 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:54245 2023-03-19 09:54:12,214 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:12,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:12,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7ef6718d22690da9d245ca8892594cd1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:54:12,267 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:54245 2023-03-19 09:54:12,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=A 2023-03-19 09:54:12,267 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:12,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:12,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=B 2023-03-19 09:54:12,267 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-19 09:54:12,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:12,267 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 145 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-19 09:54:12,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7ef6718d22690da9d245ca8892594cd1, store=C 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 4460 2023-03-19 09:54:12,268 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 4456 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1931 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5790 rows 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1918 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5753 rows 2023-03-19 09:54:12,268 INFO [Listener at localhost/45961] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-19 09:54:12,268 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:54245 2023-03-19 09:54:12,268 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:12,273 INFO [Listener at localhost/45961] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-19 09:54:12,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-19 09:54:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:12,289 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219652288"}]},"ts":"1679219652288"} 2023-03-19 09:54:12,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-19 09:54:12,290 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-19 09:54:12,292 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-19 09:54:12,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7ef6718d22690da9d245ca8892594cd1, UNASSIGN}] 2023-03-19 09:54:12,295 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7ef6718d22690da9d245ca8892594cd1, UNASSIGN 2023-03-19 09:54:12,296 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7ef6718d22690da9d245ca8892594cd1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:12,296 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219652296"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219652296"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219652296"}]},"ts":"1679219652296"} 2023-03-19 09:54:12,298 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:54:12,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-19 09:54:12,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:12,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7ef6718d22690da9d245ca8892594cd1, disabling compactions & flushes 2023-03-19 09:54:12,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:12,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-19 09:54:12,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/52989896f63040ef851b75c96afd1e39 2023-03-19 09:54:12,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-19 09:54:13,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/19d5d63e202a409b91f7d9fb523f47d3 2023-03-19 09:54:13,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-19 09:54:13,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/7f17d89533e0415f98117797e16b1480 2023-03-19 09:54:13,519 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/A/52989896f63040ef851b75c96afd1e39 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/52989896f63040ef851b75c96afd1e39 2023-03-19 09:54:13,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/52989896f63040ef851b75c96afd1e39, entries=150, sequenceid=557, filesize=11.9 K 2023-03-19 09:54:13,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/B/19d5d63e202a409b91f7d9fb523f47d3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/19d5d63e202a409b91f7d9fb523f47d3 2023-03-19 09:54:13,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/19d5d63e202a409b91f7d9fb523f47d3, entries=150, sequenceid=557, filesize=11.9 K 2023-03-19 09:54:13,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/.tmp/C/7f17d89533e0415f98117797e16b1480 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7f17d89533e0415f98117797e16b1480 2023-03-19 09:54:13,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7f17d89533e0415f98117797e16b1480, entries=150, sequenceid=557, filesize=11.9 K 2023-03-19 09:54:13,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 7ef6718d22690da9d245ca8892594cd1 in 1270ms, sequenceid=557, compaction requested=true 2023-03-19 09:54:13,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:13,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:13,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:13,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:13,537 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:54:13,537 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:54:13,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. after waiting 0 ms 2023-03-19 09:54:13,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:13,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:13,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:13,537 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. because compaction request was cancelled 2023-03-19 09:54:13,557 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/be31484c1af242e6a27dfddf811a28f2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee08f8167d564a28bfdc3767566d9e3c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/472ec6fa1a4b424087a0616c21d98c4f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b44cfeb12694a10ae603d2794c082e6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ec82845d21f84092bdbeddcc679c98cc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/978224aad8bf435eb2430b7363daabcf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/20ad5c89fd6a4877948480b571dc6dd4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/708c1a3c3139478e991400a65a3ba6dc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/964364d3f3f24513b1bf45455f2827ed, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7c59af2a4301416eb9dad5081125bee5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8aac5b07628a46849cbe916fa0fb1c06, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/382630546d1e445aac160c34686008da, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3f519429470f4843983b917635f4a9a7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1da3b6f8126840a4b7de43fe705aea4d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8cef5e5fabb94cb3baeef22108b314fd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8b13177fa0ef4322886e079f6520168f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/560ebda97f4445b2bceba5ef80717dad, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c2c1ef10a24f40ffba3be1b04d060146, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9a001e35c8d5430697b7b18c5bd115e9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/b1fe2338b4ba4cce802928a022969e9f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b3e21b0627747b88a52fa86e39cbca2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/5a7aa9ceda304597a364349b1b25ee67, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/511ad60171d64efca65ec1252e9312a8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3b88cfa0384e42389821fa728f03b177, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1ad0c1da7fff4e969b2fdd6ba314d29d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7caf952d8b29487494715ce2bd0c631d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4030da20060a4d609a7c5090914c71cb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3eadccb8a4844df485cfcd4088d05911, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9c6bb985f85c4321abd6524697dd8209, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/bd0b39e8d61649959f728c23c66a8fcd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee4e9b9478c545ea8bf003d6e4b92402, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/93dc8461742f4288850df7e66adbd065, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/cb25c55f14f947259444c237c2d4561a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/68bbc3d2dd384093bf7202d64bfdb853, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/61b9c4714dfa4fae841b0cb42905ba50, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/2df05bff1ed6480faa069a52524d8080] to archive 2023-03-19 09:54:13,559 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:54:13,564 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/be31484c1af242e6a27dfddf811a28f2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/be31484c1af242e6a27dfddf811a28f2 2023-03-19 09:54:13,565 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee08f8167d564a28bfdc3767566d9e3c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee08f8167d564a28bfdc3767566d9e3c 2023-03-19 09:54:13,567 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/472ec6fa1a4b424087a0616c21d98c4f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/472ec6fa1a4b424087a0616c21d98c4f 2023-03-19 09:54:13,568 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b44cfeb12694a10ae603d2794c082e6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b44cfeb12694a10ae603d2794c082e6 2023-03-19 09:54:13,569 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ec82845d21f84092bdbeddcc679c98cc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ec82845d21f84092bdbeddcc679c98cc 2023-03-19 09:54:13,571 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/978224aad8bf435eb2430b7363daabcf to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/978224aad8bf435eb2430b7363daabcf 2023-03-19 09:54:13,572 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/20ad5c89fd6a4877948480b571dc6dd4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/20ad5c89fd6a4877948480b571dc6dd4 2023-03-19 09:54:13,573 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/708c1a3c3139478e991400a65a3ba6dc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/708c1a3c3139478e991400a65a3ba6dc 2023-03-19 09:54:13,574 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/964364d3f3f24513b1bf45455f2827ed to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/964364d3f3f24513b1bf45455f2827ed 2023-03-19 09:54:13,575 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7c59af2a4301416eb9dad5081125bee5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7c59af2a4301416eb9dad5081125bee5 2023-03-19 09:54:13,576 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8aac5b07628a46849cbe916fa0fb1c06 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8aac5b07628a46849cbe916fa0fb1c06 2023-03-19 09:54:13,577 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/382630546d1e445aac160c34686008da to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/382630546d1e445aac160c34686008da 2023-03-19 09:54:13,578 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3f519429470f4843983b917635f4a9a7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3f519429470f4843983b917635f4a9a7 2023-03-19 09:54:13,580 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1da3b6f8126840a4b7de43fe705aea4d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1da3b6f8126840a4b7de43fe705aea4d 2023-03-19 09:54:13,581 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8cef5e5fabb94cb3baeef22108b314fd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8cef5e5fabb94cb3baeef22108b314fd 2023-03-19 09:54:13,582 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8b13177fa0ef4322886e079f6520168f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/8b13177fa0ef4322886e079f6520168f 2023-03-19 09:54:13,583 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/560ebda97f4445b2bceba5ef80717dad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/560ebda97f4445b2bceba5ef80717dad 2023-03-19 09:54:13,584 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c2c1ef10a24f40ffba3be1b04d060146 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c2c1ef10a24f40ffba3be1b04d060146 2023-03-19 09:54:13,585 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9a001e35c8d5430697b7b18c5bd115e9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9a001e35c8d5430697b7b18c5bd115e9 2023-03-19 09:54:13,586 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/b1fe2338b4ba4cce802928a022969e9f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/b1fe2338b4ba4cce802928a022969e9f 2023-03-19 09:54:13,587 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b3e21b0627747b88a52fa86e39cbca2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4b3e21b0627747b88a52fa86e39cbca2 2023-03-19 09:54:13,588 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/5a7aa9ceda304597a364349b1b25ee67 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/5a7aa9ceda304597a364349b1b25ee67 2023-03-19 09:54:13,589 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/511ad60171d64efca65ec1252e9312a8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/511ad60171d64efca65ec1252e9312a8 2023-03-19 09:54:13,591 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3b88cfa0384e42389821fa728f03b177 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3b88cfa0384e42389821fa728f03b177 2023-03-19 09:54:13,592 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1ad0c1da7fff4e969b2fdd6ba314d29d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1ad0c1da7fff4e969b2fdd6ba314d29d 2023-03-19 09:54:13,593 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7caf952d8b29487494715ce2bd0c631d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/7caf952d8b29487494715ce2bd0c631d 2023-03-19 09:54:13,594 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4030da20060a4d609a7c5090914c71cb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/4030da20060a4d609a7c5090914c71cb 2023-03-19 09:54:13,595 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3eadccb8a4844df485cfcd4088d05911 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/3eadccb8a4844df485cfcd4088d05911 2023-03-19 09:54:13,596 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9c6bb985f85c4321abd6524697dd8209 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/9c6bb985f85c4321abd6524697dd8209 2023-03-19 09:54:13,598 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/bd0b39e8d61649959f728c23c66a8fcd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/bd0b39e8d61649959f728c23c66a8fcd 2023-03-19 09:54:13,599 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee4e9b9478c545ea8bf003d6e4b92402 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/ee4e9b9478c545ea8bf003d6e4b92402 2023-03-19 09:54:13,600 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/93dc8461742f4288850df7e66adbd065 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/93dc8461742f4288850df7e66adbd065 2023-03-19 09:54:13,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/cb25c55f14f947259444c237c2d4561a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/cb25c55f14f947259444c237c2d4561a 2023-03-19 09:54:13,602 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/68bbc3d2dd384093bf7202d64bfdb853 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/68bbc3d2dd384093bf7202d64bfdb853 2023-03-19 09:54:13,603 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/61b9c4714dfa4fae841b0cb42905ba50 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/61b9c4714dfa4fae841b0cb42905ba50 2023-03-19 09:54:13,604 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/2df05bff1ed6480faa069a52524d8080 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/2df05bff1ed6480faa069a52524d8080 2023-03-19 09:54:13,629 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0971a8aeb95043b2b6296fbbed686d26, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c9b697debfe142d192a68eba4e921070, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/94be782b596d48afa07b99d7b0b87abb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c4c8488c8eed437b91f20c3287a02a49, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4828c7eb5c58409fb87eb69010409e2e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/95ded8d2b1b34c1eb87bf504fbc397c1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/2d587ae5dfd9474cb9e225ff2b65ccca, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/37e5b9e205724c4496785bada70a7d39, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/7ee62272798b476f97cf192da313aa6a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0f4e155b34eb4804ab724838a6e2131a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4fd8ffe13974406b871e291467124e91, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0843f42708344af29ce3f77cdd871579, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/d5c7bbc01f3844cb99b887183b9c1d93, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/058fdd26bd4f4e2fb01656028d3f1c0e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5f727232a9c54179aafd72a8fbab253f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/bcd50886789649e89bf26a7f0781005e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/725b5dbcc05c47cca7a3d6b617833e68, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/75ee3cb4a6c14065989e2814fcf1f9f2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/9a0b813f66684cdd9783981c44cbf367, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b0e5394317d24e58878913b878bc2bff, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/a9c09061f4074a32a0b914aef88af512, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/eb07496d2d5945c589abecc962812408, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/069bea8f256f4d1bb9d8de3b47e008b7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/68b9791432364f68964a53a898966f76, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa6b1f7c71814713871af1f7e7d3461c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5d9830de6c994abb935fc884721c137d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/1e827bc0451c4e2a870411b7026af716, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/616fcebc9eca4b2cb9f2d6b38bcacb29, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/de46489d819d4925bf292bd522199492, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b55ae9832543470394bd50801fd1003e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/cc35d17803d84853a900069c472aabe0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f6b4f1d2b0824247a46c41168417b5ea, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa04e1534bb84f2c908003135a4eaff1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/07d926b9b6244fe4b3af06d76d45968b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/770df38fee1b4ede9b189b6956f48971, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f9ef7089d10d4895832d3cb26d881227] to archive 2023-03-19 09:54:13,630 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:54:13,632 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0971a8aeb95043b2b6296fbbed686d26 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0971a8aeb95043b2b6296fbbed686d26 2023-03-19 09:54:13,633 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c9b697debfe142d192a68eba4e921070 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c9b697debfe142d192a68eba4e921070 2023-03-19 09:54:13,634 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/94be782b596d48afa07b99d7b0b87abb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/94be782b596d48afa07b99d7b0b87abb 2023-03-19 09:54:13,635 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c4c8488c8eed437b91f20c3287a02a49 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/c4c8488c8eed437b91f20c3287a02a49 2023-03-19 09:54:13,636 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4828c7eb5c58409fb87eb69010409e2e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4828c7eb5c58409fb87eb69010409e2e 2023-03-19 09:54:13,637 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/95ded8d2b1b34c1eb87bf504fbc397c1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/95ded8d2b1b34c1eb87bf504fbc397c1 2023-03-19 09:54:13,638 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/2d587ae5dfd9474cb9e225ff2b65ccca to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/2d587ae5dfd9474cb9e225ff2b65ccca 2023-03-19 09:54:13,639 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/37e5b9e205724c4496785bada70a7d39 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/37e5b9e205724c4496785bada70a7d39 2023-03-19 09:54:13,640 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/7ee62272798b476f97cf192da313aa6a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/7ee62272798b476f97cf192da313aa6a 2023-03-19 09:54:13,642 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0f4e155b34eb4804ab724838a6e2131a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0f4e155b34eb4804ab724838a6e2131a 2023-03-19 09:54:13,643 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4fd8ffe13974406b871e291467124e91 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/4fd8ffe13974406b871e291467124e91 2023-03-19 09:54:13,644 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0843f42708344af29ce3f77cdd871579 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/0843f42708344af29ce3f77cdd871579 2023-03-19 09:54:13,645 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/d5c7bbc01f3844cb99b887183b9c1d93 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/d5c7bbc01f3844cb99b887183b9c1d93 2023-03-19 09:54:13,646 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/058fdd26bd4f4e2fb01656028d3f1c0e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/058fdd26bd4f4e2fb01656028d3f1c0e 2023-03-19 09:54:13,647 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5f727232a9c54179aafd72a8fbab253f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5f727232a9c54179aafd72a8fbab253f 2023-03-19 09:54:13,648 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/bcd50886789649e89bf26a7f0781005e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/bcd50886789649e89bf26a7f0781005e 2023-03-19 09:54:13,649 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/725b5dbcc05c47cca7a3d6b617833e68 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/725b5dbcc05c47cca7a3d6b617833e68 2023-03-19 09:54:13,650 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/75ee3cb4a6c14065989e2814fcf1f9f2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/75ee3cb4a6c14065989e2814fcf1f9f2 2023-03-19 09:54:13,651 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/9a0b813f66684cdd9783981c44cbf367 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/9a0b813f66684cdd9783981c44cbf367 2023-03-19 09:54:13,652 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b0e5394317d24e58878913b878bc2bff to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b0e5394317d24e58878913b878bc2bff 2023-03-19 09:54:13,653 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/a9c09061f4074a32a0b914aef88af512 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/a9c09061f4074a32a0b914aef88af512 2023-03-19 09:54:13,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/eb07496d2d5945c589abecc962812408 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/eb07496d2d5945c589abecc962812408 2023-03-19 09:54:13,656 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/069bea8f256f4d1bb9d8de3b47e008b7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/069bea8f256f4d1bb9d8de3b47e008b7 2023-03-19 09:54:13,657 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/68b9791432364f68964a53a898966f76 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/68b9791432364f68964a53a898966f76 2023-03-19 09:54:13,658 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa6b1f7c71814713871af1f7e7d3461c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa6b1f7c71814713871af1f7e7d3461c 2023-03-19 09:54:13,659 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5d9830de6c994abb935fc884721c137d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/5d9830de6c994abb935fc884721c137d 2023-03-19 09:54:13,660 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/1e827bc0451c4e2a870411b7026af716 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/1e827bc0451c4e2a870411b7026af716 2023-03-19 09:54:13,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/616fcebc9eca4b2cb9f2d6b38bcacb29 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/616fcebc9eca4b2cb9f2d6b38bcacb29 2023-03-19 09:54:13,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/de46489d819d4925bf292bd522199492 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/de46489d819d4925bf292bd522199492 2023-03-19 09:54:13,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b55ae9832543470394bd50801fd1003e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/b55ae9832543470394bd50801fd1003e 2023-03-19 09:54:13,664 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/cc35d17803d84853a900069c472aabe0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/cc35d17803d84853a900069c472aabe0 2023-03-19 09:54:13,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f6b4f1d2b0824247a46c41168417b5ea to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f6b4f1d2b0824247a46c41168417b5ea 2023-03-19 09:54:13,667 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa04e1534bb84f2c908003135a4eaff1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/aa04e1534bb84f2c908003135a4eaff1 2023-03-19 09:54:13,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/07d926b9b6244fe4b3af06d76d45968b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/07d926b9b6244fe4b3af06d76d45968b 2023-03-19 09:54:13,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/770df38fee1b4ede9b189b6956f48971 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/770df38fee1b4ede9b189b6956f48971 2023-03-19 09:54:13,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f9ef7089d10d4895832d3cb26d881227 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/f9ef7089d10d4895832d3cb26d881227 2023-03-19 09:54:13,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/727400a903fa44fbb903e50130528b4c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/4b6ddfa05e604a1ebb14026680c4f28f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0b5a2a56cdad4842bf544664c4f17fc1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/34c38bdc6cb946818f8f57bb89d20dd5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e8f2f0272cfa4299bb2e3765ac39417e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7038ff2871364e1aacbc4d4d0855c506, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/382312e6aa9142fc89e7554e38bae254, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/eb1b0e215ebb4ae2819ab27b01f47fc5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/63c993707c034e6a8befcb051dcb4812, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/3e5272e7a8934e1d850f3b9c8bc22eaa, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/d7c90852426c4872a04d4d877c41f01a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7cebd238c36e4e168bf44430894b7e1a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e4f14638552e49129c2542e14b8bf6e4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/818a1d5cd1c54969ace69a1fcc7778ab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/dc0d7e5799ed427eb57ab23678424f63, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/69d08d9508af4536939a1f2d21239c3c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/9984520baca44d5f8ef0264f0a4fa5fc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/071d60ce5ebc4c74875c18545fce22af, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/747e7f7d63e44ac2a0e19935c97909e6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/ccf5803a8c444ec3b3cf8b0714e3f013, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/c850cd4dbe194818b75aa523c426306f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/8eb48457692446a48a4f20507614e714, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/cee6b6a5c0554356960351aa55e65ed2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/30a44c5621784425ac7f0f691788176b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a02b8982d1c34beeaab58b23198e2c5e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/2ee1dc61458f4ff78a522959eec2bf27, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/b02234d6d6e444e2b61278b8c4739ea8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a7bb78c9abee4403b235c1880937688e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/32edc451b08b4cee81d7419361a61257, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/90e0b4d83fa44e57be42d8904e647444, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0c28475f2a5f455b9cc82f7185645255, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/23d3956a4b2f40968768f28270f3f29e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/12ce4e06ce474517a41c77e585caf1bd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/96775c4d4cf1469e887a4af50a179999] to archive 2023-03-19 09:54:13,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:54:13,677 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/727400a903fa44fbb903e50130528b4c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/727400a903fa44fbb903e50130528b4c 2023-03-19 09:54:13,678 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/4b6ddfa05e604a1ebb14026680c4f28f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/4b6ddfa05e604a1ebb14026680c4f28f 2023-03-19 09:54:13,679 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0b5a2a56cdad4842bf544664c4f17fc1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0b5a2a56cdad4842bf544664c4f17fc1 2023-03-19 09:54:13,680 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/34c38bdc6cb946818f8f57bb89d20dd5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/34c38bdc6cb946818f8f57bb89d20dd5 2023-03-19 09:54:13,682 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e8f2f0272cfa4299bb2e3765ac39417e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e8f2f0272cfa4299bb2e3765ac39417e 2023-03-19 09:54:13,683 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7038ff2871364e1aacbc4d4d0855c506 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7038ff2871364e1aacbc4d4d0855c506 2023-03-19 09:54:13,684 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/382312e6aa9142fc89e7554e38bae254 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/382312e6aa9142fc89e7554e38bae254 2023-03-19 09:54:13,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/eb1b0e215ebb4ae2819ab27b01f47fc5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/eb1b0e215ebb4ae2819ab27b01f47fc5 2023-03-19 09:54:13,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/63c993707c034e6a8befcb051dcb4812 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/63c993707c034e6a8befcb051dcb4812 2023-03-19 09:54:13,687 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/3e5272e7a8934e1d850f3b9c8bc22eaa to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/3e5272e7a8934e1d850f3b9c8bc22eaa 2023-03-19 09:54:13,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/d7c90852426c4872a04d4d877c41f01a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/d7c90852426c4872a04d4d877c41f01a 2023-03-19 09:54:13,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7cebd238c36e4e168bf44430894b7e1a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7cebd238c36e4e168bf44430894b7e1a 2023-03-19 09:54:13,690 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e4f14638552e49129c2542e14b8bf6e4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/e4f14638552e49129c2542e14b8bf6e4 2023-03-19 09:54:13,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/818a1d5cd1c54969ace69a1fcc7778ab to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/818a1d5cd1c54969ace69a1fcc7778ab 2023-03-19 09:54:13,692 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/dc0d7e5799ed427eb57ab23678424f63 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/dc0d7e5799ed427eb57ab23678424f63 2023-03-19 09:54:13,694 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/69d08d9508af4536939a1f2d21239c3c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/69d08d9508af4536939a1f2d21239c3c 2023-03-19 09:54:13,695 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/9984520baca44d5f8ef0264f0a4fa5fc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/9984520baca44d5f8ef0264f0a4fa5fc 2023-03-19 09:54:13,696 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/071d60ce5ebc4c74875c18545fce22af to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/071d60ce5ebc4c74875c18545fce22af 2023-03-19 09:54:13,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/747e7f7d63e44ac2a0e19935c97909e6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/747e7f7d63e44ac2a0e19935c97909e6 2023-03-19 09:54:13,699 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/ccf5803a8c444ec3b3cf8b0714e3f013 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/ccf5803a8c444ec3b3cf8b0714e3f013 2023-03-19 09:54:13,700 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/c850cd4dbe194818b75aa523c426306f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/c850cd4dbe194818b75aa523c426306f 2023-03-19 09:54:13,701 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/8eb48457692446a48a4f20507614e714 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/8eb48457692446a48a4f20507614e714 2023-03-19 09:54:13,702 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/cee6b6a5c0554356960351aa55e65ed2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/cee6b6a5c0554356960351aa55e65ed2 2023-03-19 09:54:13,703 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/30a44c5621784425ac7f0f691788176b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/30a44c5621784425ac7f0f691788176b 2023-03-19 09:54:13,704 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a02b8982d1c34beeaab58b23198e2c5e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a02b8982d1c34beeaab58b23198e2c5e 2023-03-19 09:54:13,705 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/2ee1dc61458f4ff78a522959eec2bf27 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/2ee1dc61458f4ff78a522959eec2bf27 2023-03-19 09:54:13,707 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/b02234d6d6e444e2b61278b8c4739ea8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/b02234d6d6e444e2b61278b8c4739ea8 2023-03-19 09:54:13,708 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a7bb78c9abee4403b235c1880937688e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/a7bb78c9abee4403b235c1880937688e 2023-03-19 09:54:13,709 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/32edc451b08b4cee81d7419361a61257 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/32edc451b08b4cee81d7419361a61257 2023-03-19 09:54:13,710 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/90e0b4d83fa44e57be42d8904e647444 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/90e0b4d83fa44e57be42d8904e647444 2023-03-19 09:54:13,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0c28475f2a5f455b9cc82f7185645255 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/0c28475f2a5f455b9cc82f7185645255 2023-03-19 09:54:13,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/23d3956a4b2f40968768f28270f3f29e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/23d3956a4b2f40968768f28270f3f29e 2023-03-19 09:54:13,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/12ce4e06ce474517a41c77e585caf1bd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/12ce4e06ce474517a41c77e585caf1bd 2023-03-19 09:54:13,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/96775c4d4cf1469e887a4af50a179999 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/96775c4d4cf1469e887a4af50a179999 2023-03-19 09:54:13,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/recovered.edits/560.seqid, newMaxSeqId=560, maxSeqId=1 2023-03-19 09:54:13,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1. 2023-03-19 09:54:13,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7ef6718d22690da9d245ca8892594cd1: 2023-03-19 09:54:13,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:13,723 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7ef6718d22690da9d245ca8892594cd1, regionState=CLOSED 2023-03-19 09:54:13,723 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219653723"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219653723"}]},"ts":"1679219653723"} 2023-03-19 09:54:13,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-19 09:54:13,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 7ef6718d22690da9d245ca8892594cd1, server=jenkins-hbase4.apache.org,34669,1679219618625 in 1.4260 sec 2023-03-19 09:54:13,728 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-19 09:54:13,728 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7ef6718d22690da9d245ca8892594cd1, UNASSIGN in 1.4320 sec 2023-03-19 09:54:13,728 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219653728"}]},"ts":"1679219653728"} 2023-03-19 09:54:13,729 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-19 09:54:13,731 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-19 09:54:13,733 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4510 sec 2023-03-19 09:54:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-19 09:54:14,394 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-19 09:54:14,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-19 09:54:14,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:14,408 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:14,409 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:14,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-19 09:54:14,413 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:14,416 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/recovered.edits] 2023-03-19 09:54:14,419 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1bb2657de2e94f689203361df7747974 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/1bb2657de2e94f689203361df7747974 2023-03-19 09:54:14,422 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/52989896f63040ef851b75c96afd1e39 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/52989896f63040ef851b75c96afd1e39 2023-03-19 09:54:14,423 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c71ef7768bdc48798bf9463e5a8ac338 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/A/c71ef7768bdc48798bf9463e5a8ac338 2023-03-19 09:54:14,426 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/19d5d63e202a409b91f7d9fb523f47d3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/19d5d63e202a409b91f7d9fb523f47d3 2023-03-19 09:54:14,428 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/8465f12688aa4c98a6360ff3da39b67d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/8465f12688aa4c98a6360ff3da39b67d 2023-03-19 09:54:14,430 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/8d5c0d5eaf3d46c883a922c4155994da to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/B/8d5c0d5eaf3d46c883a922c4155994da 2023-03-19 09:54:14,432 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7f17d89533e0415f98117797e16b1480 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/7f17d89533e0415f98117797e16b1480 2023-03-19 09:54:14,433 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/80dbeca409a943588af85bbfd69a998f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/80dbeca409a943588af85bbfd69a998f 2023-03-19 09:54:14,434 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/aa120617d146424b8ef7af7c4115fe66 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/C/aa120617d146424b8ef7af7c4115fe66 2023-03-19 09:54:14,437 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/recovered.edits/560.seqid to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1/recovered.edits/560.seqid 2023-03-19 09:54:14,438 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/7ef6718d22690da9d245ca8892594cd1 2023-03-19 09:54:14,438 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:54:14,440 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:14,460 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-19 09:54:14,503 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-19 09:54:14,504 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:14,504 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-19 09:54:14,504 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679219654504"}]},"ts":"9223372036854775807"} 2023-03-19 09:54:14,507 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-19 09:54:14,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7ef6718d22690da9d245ca8892594cd1, NAME => 'TestAcidGuarantees,,1679219622169.7ef6718d22690da9d245ca8892594cd1.', STARTKEY => '', ENDKEY => ''}] 2023-03-19 09:54:14,507 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-19 09:54:14,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679219654507"}]},"ts":"9223372036854775807"} 2023-03-19 09:54:14,509 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-19 09:54:14,511 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:14,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-19 09:54:14,512 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 110 msec 2023-03-19 09:54:14,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-19 09:54:14,713 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-19 09:54:14,730 INFO [Listener at localhost/45961] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d86bf75-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d86bf75-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-413854900_15 at /127.0.0.1:55272 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:34669-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d86bf75-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d86bf75-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=530 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=445 (was 395) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1840 (was 2525) 2023-03-19 09:54:14,741 INFO [Listener at localhost/45961] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=530, MaxFileDescriptor=60000, SystemLoadAverage=445, ProcessCount=169, AvailableMemoryMB=1839 2023-03-19 09:54:14,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:54:14,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:54:14,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:14,746 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-19 09:54:14,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-19 09:54:14,747 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-19 09:54:14,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-19 09:54:14,748 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:14,749 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 empty. 2023-03-19 09:54:14,749 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:14,749 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:54:14,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-19 09:54:15,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-19 09:54:15,166 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:54:15,167 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4a5125924c57b3613b6b6094511223b4, NAME => 'TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp 2023-03-19 09:54:15,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-19 09:54:15,574 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:54:15,574 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4a5125924c57b3613b6b6094511223b4, disabling compactions & flushes 2023-03-19 09:54:15,574 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,574 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,574 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. after waiting 0 ms 2023-03-19 09:54:15,574 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,574 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,574 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:15,577 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-19 09:54:15,577 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219655577"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219655577"}]},"ts":"1679219655577"} 2023-03-19 09:54:15,579 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-19 09:54:15,580 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-19 09:54:15,580 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219655580"}]},"ts":"1679219655580"} 2023-03-19 09:54:15,581 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-19 09:54:15,587 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, ASSIGN}] 2023-03-19 09:54:15,588 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, ASSIGN 2023-03-19 09:54:15,589 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=false 2023-03-19 09:54:15,740 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:15,740 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219655740"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219655740"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219655740"}]},"ts":"1679219655740"} 2023-03-19 09:54:15,742 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:54:15,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-19 09:54:15,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4a5125924c57b3613b6b6094511223b4, NAME => 'TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:54:15,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:54:15,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,899 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,901 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A 2023-03-19 09:54:15,901 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A 2023-03-19 09:54:15,901 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:15,901 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4a5125924c57b3613b6b6094511223b4 columnFamilyName A 2023-03-19 09:54:15,902 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(310): Store=4a5125924c57b3613b6b6094511223b4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:15,902 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,903 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B 2023-03-19 09:54:15,903 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B 2023-03-19 09:54:15,903 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:15,903 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4a5125924c57b3613b6b6094511223b4 columnFamilyName B 2023-03-19 09:54:15,904 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(310): Store=4a5125924c57b3613b6b6094511223b4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:15,904 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,905 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C 2023-03-19 09:54:15,905 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C 2023-03-19 09:54:15,905 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:15,905 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4a5125924c57b3613b6b6094511223b4 columnFamilyName C 2023-03-19 09:54:15,906 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(310): Store=4a5125924c57b3613b6b6094511223b4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:15,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:54:15,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:15,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:54:15,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4a5125924c57b3613b6b6094511223b4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69423295, jitterRate=0.034487709403038025}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:54:15,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:15,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., pid=18, masterSystemTime=1679219655893 2023-03-19 09:54:15,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:15,914 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:15,914 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219655914"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219655914"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219655914"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219655914"}]},"ts":"1679219655914"} 2023-03-19 09:54:15,917 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-19 09:54:15,917 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 in 173 msec 2023-03-19 09:54:15,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-19 09:54:15,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, ASSIGN in 331 msec 2023-03-19 09:54:15,920 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-19 09:54:15,920 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219655920"}]},"ts":"1679219655920"} 2023-03-19 09:54:15,921 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-19 09:54:15,923 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-19 09:54:15,924 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1800 sec 2023-03-19 09:54:16,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-19 09:54:16,852 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-19 09:54:16,853 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x5b8d0eb2 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:16,858 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52274650, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:16,859 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:16,863 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36078, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:16,864 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-19 09:54:16,866 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34636, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-19 09:54:16,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:54:16,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:54:16,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:17,293 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-19 09:54:17,294 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:54:17,294 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-19 09:54:17,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-19 09:54:17,310 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, REOPEN/MOVE}] 2023-03-19 09:54:17,311 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, REOPEN/MOVE 2023-03-19 09:54:17,312 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:17,312 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219657311"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219657311"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219657311"}]},"ts":"1679219657311"} 2023-03-19 09:54:17,313 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:54:17,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4a5125924c57b3613b6b6094511223b4, disabling compactions & flushes 2023-03-19 09:54:17,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. after waiting 0 ms 2023-03-19 09:54:17,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-19 09:54:17,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:17,470 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 4a5125924c57b3613b6b6094511223b4 to self. 2023-03-19 09:54:17,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,472 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=CLOSED 2023-03-19 09:54:17,472 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219657472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219657472"}]},"ts":"1679219657472"} 2023-03-19 09:54:17,475 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-19 09:54:17,475 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 in 160 msec 2023-03-19 09:54:17,476 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=true 2023-03-19 09:54:17,626 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:17,626 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219657626"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219657626"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219657626"}]},"ts":"1679219657626"} 2023-03-19 09:54:17,628 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:54:17,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4a5125924c57b3613b6b6094511223b4, NAME => 'TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:54:17,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:54:17,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,786 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,787 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A 2023-03-19 09:54:17,787 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A 2023-03-19 09:54:17,787 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:17,790 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4a5125924c57b3613b6b6094511223b4 columnFamilyName A 2023-03-19 09:54:17,792 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(310): Store=4a5125924c57b3613b6b6094511223b4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:17,792 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,793 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B 2023-03-19 09:54:17,793 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B 2023-03-19 09:54:17,794 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:17,794 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4a5125924c57b3613b6b6094511223b4 columnFamilyName B 2023-03-19 09:54:17,795 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(310): Store=4a5125924c57b3613b6b6094511223b4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:17,795 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,795 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C 2023-03-19 09:54:17,796 DEBUG [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C 2023-03-19 09:54:17,796 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:17,796 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4a5125924c57b3613b6b6094511223b4 columnFamilyName C 2023-03-19 09:54:17,797 INFO [StoreOpener-4a5125924c57b3613b6b6094511223b4-1] regionserver.HStore(310): Store=4a5125924c57b3613b6b6094511223b4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:17,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:54:17,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4a5125924c57b3613b6b6094511223b4; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66729643, jitterRate=-0.005650833249092102}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:54:17,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:17,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., pid=23, masterSystemTime=1679219657780 2023-03-19 09:54:17,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,806 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:17,806 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219657805"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219657805"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219657805"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219657805"}]},"ts":"1679219657805"} 2023-03-19 09:54:17,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-19 09:54:17,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 in 179 msec 2023-03-19 09:54:17,810 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-19 09:54:17,810 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, REOPEN/MOVE in 499 msec 2023-03-19 09:54:17,813 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-19 09:54:17,813 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 515 msec 2023-03-19 09:54:17,815 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 938 msec 2023-03-19 09:54:17,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-19 09:54:17,824 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x051c62b6 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,829 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21281a37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,830 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,834 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bab6b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,834 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x21265fcb to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,837 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24bd3872, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,838 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,843 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,843 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,848 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,848 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,852 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,853 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,856 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,856 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,859 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,860 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x58e27a32 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:17,864 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a7b66a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:17,871 DEBUG [hconnection-0x4fe6e4ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,872 DEBUG [hconnection-0x6b6226c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:17,872 DEBUG [hconnection-0x5d7da072-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,873 DEBUG [hconnection-0x12051855-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,873 DEBUG [hconnection-0x5b9818f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,874 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36092, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,874 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36122, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,876 DEBUG [hconnection-0x69d4ad36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,884 DEBUG [hconnection-0x6ff4300e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,884 DEBUG [hconnection-0x4390ee54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,884 DEBUG [hconnection-0xd8059ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:17,884 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,884 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36094, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,885 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36132, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,885 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36134, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,886 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36142, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,889 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36144, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,889 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36148, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:17,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:17,899 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:17,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:17,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:17,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:17,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:17,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:17,902 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:17,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:17,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:17,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:17,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:17,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:17,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:17,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:17,913 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:17,913 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:17,913 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:17,913 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:17,914 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:17,914 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:17,914 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:17,914 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-19 09:54:17,945 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:17,945 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:17,945 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:17,945 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:17,945 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:17,945 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:17,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:17,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:17,983 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303197b91f94d19cf471e9c0a04db6493eaad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197b91f94d19cf471e9c0a04db6493eaad 2023-03-19 09:54:17,991 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=11, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/7d8c1fbea1cf426a9ddffd408cfd66ee 2023-03-19 09:54:18,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679219657962, max timestamp=1679219658027Num uniques 859; 2023-03-19 09:54:18,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-19 09:54:18,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679219657962, max timestamp=1679219658027Num uniques 859; 2023-03-19 09:54:18,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-19 09:54:18,033 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/785cbbfae5e847778a14d8af7c722f46 2023-03-19 09:54:18,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219658017, max timestamp=1679219658027Num uniques 859; 2023-03-19 09:54:18,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:18,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219718052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219718056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219718054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219718060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219718060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,080 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/1f956f5e701e430181210f5e9c3d288a 2023-03-19 09:54:18,090 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/7d8c1fbea1cf426a9ddffd408cfd66ee as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d8c1fbea1cf426a9ddffd408cfd66ee 2023-03-19 09:54:18,095 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d8c1fbea1cf426a9ddffd408cfd66ee, entries=100, sequenceid=11, filesize=18.4 K 2023-03-19 09:54:18,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/785cbbfae5e847778a14d8af7c722f46 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/785cbbfae5e847778a14d8af7c722f46 2023-03-19 09:54:18,112 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/785cbbfae5e847778a14d8af7c722f46, entries=100, sequenceid=11, filesize=9.3 K 2023-03-19 09:54:18,116 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/1f956f5e701e430181210f5e9c3d288a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1f956f5e701e430181210f5e9c3d288a 2023-03-19 09:54:18,122 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1f956f5e701e430181210f5e9c3d288a, entries=100, sequenceid=11, filesize=9.3 K 2023-03-19 09:54:18,123 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 4a5125924c57b3613b6b6094511223b4 in 209ms, sequenceid=11, compaction requested=false 2023-03-19 09:54:18,123 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:18,123 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:18,125 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,126 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45037be9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:18,133 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:18,133 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,134 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:18,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:18,136 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:18,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:18,143 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:18,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:18,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:18,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:18,143 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:18,150 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,150 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,150 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:18,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:18,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:18,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,157 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,158 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:18,158 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:18,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219658054, max timestamp=1679219658165Num uniques 1318; 2023-03-19 09:54:18,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=60358, new segment data size=7688, suffix heap size=165040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1318, new segment cells count=168 2023-03-19 09:54:18,173 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=116.29 KB heapSize=322.67 KB 2023-03-19 09:54:18,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:18,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219658054, max timestamp=1679219658170Num uniques 1318; 2023-03-19 09:54:18,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=60358, new segment data size=7688, suffix heap size=165040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1318, new segment cells count=168 2023-03-19 09:54:18,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:18,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:18,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:18,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:18,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:18,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:54:18,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219718201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219718201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219718208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219718209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219718209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,258 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:18,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:18,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:18,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:18,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:18,277 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:18,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:18,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:18,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,279 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,279 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:18,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:18,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:18,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:18,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:18,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:18,284 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,284 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:18,284 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:18,284 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,286 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:18,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791fd807[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:18,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:18,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:18,292 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,292 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:18,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:18,296 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:18,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:18,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:18,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:18,296 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:18,301 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,301 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,301 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,304 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:18,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:18,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,312 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:18,312 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:18,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:18,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219718312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219718313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219718319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219718319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219718320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,412 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:18,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:18,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:18,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:18,418 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:18,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:18,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:18,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,420 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:18,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:18,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:18,432 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,432 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:18,432 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:18,432 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,434 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7decd054[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:18,436 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:18,436 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:18,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:18,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:18,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:18,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:18,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:18,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:18,446 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:18,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:18,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:18,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:18,446 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:18,448 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,449 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,449 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,450 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:18,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:18,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,453 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,454 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:18,454 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:18,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:18,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219718516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219718519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219718521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219718521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219718522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,554 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:18,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:18,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:18,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:18,559 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:18,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:18,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:18,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,560 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,561 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,561 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:18,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:18,565 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,565 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:18,565 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:18,565 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:18,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,567 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226a33b2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:18,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:18,570 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,571 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:18,571 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,571 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:18,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,573 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:18,573 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,573 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:18,576 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:18,576 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:18,576 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:18,576 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:18,576 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:18,578 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,578 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,578 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,581 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:18,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,583 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,583 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:18,605 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303195838168af8264a4baa81e0b461a74755 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195838168af8264a4baa81e0b461a74755 2023-03-19 09:54:18,683 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:18,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:18,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:18,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:18,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:18,688 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:18,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:18,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:18,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:18,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:18,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c68c825[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:18,699 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:18,699 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,700 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:18,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:18,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:18,708 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:18,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:18,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:18,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:18,708 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:18,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,710 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,711 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:18,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,715 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:18,715 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:18,816 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:18,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:18,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:18,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:18,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:18,820 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:18,820 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,820 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:18,820 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:18,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219718820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219718822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219718824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219718825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:18,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:18,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:18,828 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,828 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:18,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:18,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219718826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:18,832 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,832 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:18,832 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:18,832 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,836 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b30f84f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:18,838 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:18,838 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:18,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:18,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,840 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:18,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:18,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:18,845 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:18,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:18,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:18,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:18,845 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:18,847 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,847 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,847 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,849 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:18,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:18,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:18,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:18,860 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:18,860 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:18,960 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:18,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:18,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:18,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:18,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:18,968 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:18,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:18,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:18,969 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,969 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,970 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:18,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:18,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:18,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:18,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:18,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:18,985 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,985 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:18,985 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:18,985 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:18,987 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:18,991 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:18,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c5f90d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:18,991 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,993 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:18,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:18,995 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:18,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:18,997 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:18,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:18,998 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:18,998 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:18,998 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:19,000 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,000 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,000 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,000 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:19,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,006 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:19,006 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:19,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:19,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,037 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/dae7bbc9546d4812bfe9664a9211deee 2023-03-19 09:54:19,106 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:19,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:19,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:19,110 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:19,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:19,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:19,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,113 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,113 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,113 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:19,115 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:19,115 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:19,115 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:19,115 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:19,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c1a104[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:19,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,120 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,120 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:19,120 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:19,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,120 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:19,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:19,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:19,124 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:19,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:19,124 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:19,127 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,127 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,127 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,128 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,135 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,135 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,135 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:19,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,235 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:19,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:19,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:19,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:19,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:19,252 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:19,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:19,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:19,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,253 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:19,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:19,256 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:19,256 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:19,256 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:19,256 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,261 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-19 09:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d27ab9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:19,263 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:19,263 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:19,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:19,270 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:19,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:19,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:19,273 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:19,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:19,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:19,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:19,273 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:19,275 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,275 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,275 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,279 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,283 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:19,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:19,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:19,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,283 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:19,283 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:19,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:19,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:19,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219719326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:19,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219719327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:19,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219719328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:19,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219719329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:19,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219719332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,383 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:19,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:19,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:19,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:19,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:19,392 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:19,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:19,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:19,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:19,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,396 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:19,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:19,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:19,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:19,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:19,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,408 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:19,411 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:19,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555c1d4c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:19,411 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,412 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:19,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:19,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:19,416 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:19,419 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:19,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:19,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:19,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:19,419 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:19,421 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,421 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,421 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:19,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,422 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,423 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,426 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,426 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:19,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:19,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:19,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:19,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:19,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:19,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,427 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:19,427 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:19,451 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/66ca74de0a964951a1e1bb998810654c 2023-03-19 09:54:19,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/991efbaeb0d7413baf9c47a27f8196a0 2023-03-19 09:54:19,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/dae7bbc9546d4812bfe9664a9211deee as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dae7bbc9546d4812bfe9664a9211deee 2023-03-19 09:54:19,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dae7bbc9546d4812bfe9664a9211deee, entries=150, sequenceid=50, filesize=25.1 K 2023-03-19 09:54:19,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/66ca74de0a964951a1e1bb998810654c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/66ca74de0a964951a1e1bb998810654c 2023-03-19 09:54:19,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/66ca74de0a964951a1e1bb998810654c, entries=150, sequenceid=50, filesize=11.6 K 2023-03-19 09:54:19,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/991efbaeb0d7413baf9c47a27f8196a0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/991efbaeb0d7413baf9c47a27f8196a0 2023-03-19 09:54:19,528 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:19,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:19,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:19,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:19,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:19,533 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:19,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:19,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:19,534 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,534 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/991efbaeb0d7413baf9c47a27f8196a0, entries=150, sequenceid=50, filesize=11.6 K 2023-03-19 09:54:19,535 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:19,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:19,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~198.67 KB/203440, currentSize=114.05 KB/116790 for 4a5125924c57b3613b6b6094511223b4 in 1362ms, sequenceid=50, compaction requested=false 2023-03-19 09:54:19,535 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:19,535 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:19,536 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:19,536 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:19,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:19,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:19,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:19,545 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:19,546 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:19,546 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-19 09:54:19,546 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:19,546 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:19,546 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:19,546 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:19,546 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:19,547 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:19,972 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319a6e79e172a9c49b5aba4196dbc4cdf67 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a6e79e172a9c49b5aba4196dbc4cdf67 2023-03-19 09:54:19,996 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/7d0f2728548749f3966200e4d3cead9f 2023-03-19 09:54:20,020 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/5a5e1b25eda14389abf66dfacc010029 2023-03-19 09:54:20,045 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/757eb84f76ec473ca0cc4d1bc75bceeb 2023-03-19 09:54:20,055 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/7d0f2728548749f3966200e4d3cead9f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d0f2728548749f3966200e4d3cead9f 2023-03-19 09:54:20,063 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d0f2728548749f3966200e4d3cead9f, entries=150, sequenceid=70, filesize=25.1 K 2023-03-19 09:54:20,064 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/5a5e1b25eda14389abf66dfacc010029 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5a5e1b25eda14389abf66dfacc010029 2023-03-19 09:54:20,069 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5a5e1b25eda14389abf66dfacc010029, entries=150, sequenceid=70, filesize=11.6 K 2023-03-19 09:54:20,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/757eb84f76ec473ca0cc4d1bc75bceeb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/757eb84f76ec473ca0cc4d1bc75bceeb 2023-03-19 09:54:20,077 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/757eb84f76ec473ca0cc4d1bc75bceeb, entries=150, sequenceid=70, filesize=11.6 K 2023-03-19 09:54:20,078 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 4a5125924c57b3613b6b6094511223b4 in 532ms, sequenceid=70, compaction requested=true 2023-03-19 09:54:20,078 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:20,079 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,080 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:20,080 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:20,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:20,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1e9b7b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:20,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:20,094 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:20,094 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:20,094 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:20,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:20,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:20,096 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:20,099 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:20,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:20,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:20,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:20,099 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:20,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:20,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:20,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:20,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,103 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:20,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:20,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:20,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:20,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:20,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:20,107 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:20,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:20,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,107 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:20,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:20,208 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:20,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:20,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:20,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:20,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:20,232 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:20,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:20,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:20,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:20,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,234 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:20,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:20,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:20,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:20,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:20,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:20,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:20,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:20,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:20,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:20,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:20,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:20,244 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449c0bd[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:20,247 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:20,247 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,249 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:20,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:20,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:20,254 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:20,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:20,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:20,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:20,254 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:20,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:20,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:20,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,257 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,259 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:20,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:20,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:20,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:20,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:20,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:20,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:20,262 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:20,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,262 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:20,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:20,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,262 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:20,362 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:20,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:20,367 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:20,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:20,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:20,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:20,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,369 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:20,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:20,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:20,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:20,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:20,372 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:20,372 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:20,372 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-19 09:54:20,373 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:20,373 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:20,373 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:20,373 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:20,373 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:20,373 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:20,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:20,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:20,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219660393, max timestamp=1679219660406Num uniques 859; 2023-03-19 09:54:20,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219660393, max timestamp=1679219660406Num uniques 859; 2023-03-19 09:54:20,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219660393, max timestamp=1679219660406Num uniques 859; 2023-03-19 09:54:20,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:20,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:20,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:20,412 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319953c8192ccba423489cf9a4df54ae9a3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319953c8192ccba423489cf9a4df54ae9a3 2023-03-19 09:54:20,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219720433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219660429, max timestamp=1679219660441Num uniques 1018; 2023-03-19 09:54:20,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:54:20,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219660429, max timestamp=1679219660441Num uniques 1018; 2023-03-19 09:54:20,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:54:20,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219660429, max timestamp=1679219660441Num uniques 1018; 2023-03-19 09:54:20,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:54:20,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219720465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219720467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219720467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219720468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219720544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219720572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219720574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219720574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219720574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219720750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219720777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219720777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219720777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:20,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219720778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:20,839 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=80, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/11921d427632417eaf32be2cb2c792c8 2023-03-19 09:54:21,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219721055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219721080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219721081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219721082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219721082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,252 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/725b7ae91a3d4e779b9924c037e04a3c 2023-03-19 09:54:21,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219721558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219721586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219721586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219721586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219721589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,663 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/bf5705e07bb9410d9ba5b213293d6acd 2023-03-19 09:54:21,669 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/11921d427632417eaf32be2cb2c792c8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/11921d427632417eaf32be2cb2c792c8 2023-03-19 09:54:21,674 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/11921d427632417eaf32be2cb2c792c8, entries=300, sequenceid=80, filesize=45.5 K 2023-03-19 09:54:21,675 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/725b7ae91a3d4e779b9924c037e04a3c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/725b7ae91a3d4e779b9924c037e04a3c 2023-03-19 09:54:21,680 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/725b7ae91a3d4e779b9924c037e04a3c, entries=150, sequenceid=80, filesize=11.6 K 2023-03-19 09:54:21,681 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/bf5705e07bb9410d9ba5b213293d6acd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/bf5705e07bb9410d9ba5b213293d6acd 2023-03-19 09:54:21,685 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/bf5705e07bb9410d9ba5b213293d6acd, entries=150, sequenceid=80, filesize=11.6 K 2023-03-19 09:54:21,686 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 4a5125924c57b3613b6b6094511223b4 in 1314ms, sequenceid=80, compaction requested=true 2023-03-19 09:54:21,686 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:21,686 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:21,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:21,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:21,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:21,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e74753f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:21,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:21,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,691 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:21,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:21,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:21,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:21,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:21,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:21,704 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:21,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:21,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:21,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:21,704 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:21,707 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,707 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:21,707 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:21,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:21,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:21,712 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:21,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:21,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:21,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,712 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:21,712 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:21,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:21,812 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:21,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:21,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:21,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:21,816 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:21,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:21,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:21,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:21,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:21,818 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:21,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:21,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:21,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:21,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:21,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:21,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:21,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:21,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:21,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:21,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:21,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:21,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:21,820 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:21,820 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:21,820 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-03-19 09:54:21,821 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:21,821 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:21,821 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:21,821 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:21,821 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:21,821 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:21,837 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319b3436008a1bd4286ae6ed7a828153c87 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b3436008a1bd4286ae6ed7a828153c87 2023-03-19 09:54:22,254 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/c55d3016429c4079bcda8236a65eda66 2023-03-19 09:54:22,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:22,571 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:22,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:22,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219722590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:22,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219722592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:22,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219722594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:22,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219722597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:22,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219722599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,670 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/019f7b4d7e61496dbb93b9fa35104a7b 2023-03-19 09:54:22,681 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/a447f8f51d81474f89c64b3878c20d49 2023-03-19 09:54:22,686 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/c55d3016429c4079bcda8236a65eda66 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c55d3016429c4079bcda8236a65eda66 2023-03-19 09:54:22,690 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c55d3016429c4079bcda8236a65eda66, entries=150, sequenceid=133, filesize=25.3 K 2023-03-19 09:54:22,691 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/019f7b4d7e61496dbb93b9fa35104a7b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/019f7b4d7e61496dbb93b9fa35104a7b 2023-03-19 09:54:22,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:22,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219722691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,697 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/019f7b4d7e61496dbb93b9fa35104a7b, entries=150, sequenceid=133, filesize=11.8 K 2023-03-19 09:54:22,698 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/a447f8f51d81474f89c64b3878c20d49 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/a447f8f51d81474f89c64b3878c20d49 2023-03-19 09:54:22,703 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/a447f8f51d81474f89c64b3878c20d49, entries=150, sequenceid=133, filesize=11.8 K 2023-03-19 09:54:22,703 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for 4a5125924c57b3613b6b6094511223b4 in 883ms, sequenceid=133, compaction requested=true 2023-03-19 09:54:22,704 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:22,704 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:22,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,705 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:22,706 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,706 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf7b9c0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:22,707 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:22,707 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,710 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:22,710 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,710 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:22,711 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:22,714 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:22,714 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:22,714 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:22,714 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:22,714 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:22,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:22,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:22,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:22,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:22,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:22,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:22,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,722 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:22,722 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:22,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:22,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:22,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:22,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:22,822 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:22,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:22,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:22,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:22,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:22,828 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:22,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:22,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:22,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:22,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:22,831 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:22,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:22,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:22,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:22,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:22,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:22,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:22,833 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:22,833 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:22,833 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-19 09:54:22,833 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:22,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:22,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:22,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:22,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:22,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:22,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:22,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:22,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219662925, max timestamp=1679219662967Num uniques 859; 2023-03-19 09:54:22,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219662925, max timestamp=1679219662967Num uniques 859; 2023-03-19 09:54:22,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:22,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219662925, max timestamp=1679219662967Num uniques 859; 2023-03-19 09:54:22,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:22,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:23,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679219662996, max timestamp=1679219663037Num uniques 1018; 2023-03-19 09:54:23,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679219662996, max timestamp=1679219663037Num uniques 1018; 2023-03-19 09:54:23,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-19 09:54:23,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679219662996, max timestamp=1679219663037Num uniques 1018; 2023-03-19 09:54:23,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-19 09:54:23,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-19 09:54:23,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:23,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219723101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:23,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219723202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,249 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319c36a8030db4b44c59c0af924b3440ce9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319c36a8030db4b44c59c0af924b3440ce9 2023-03-19 09:54:23,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:23,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219723406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,654 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/a4bf8d97226047199d56ee1df996bcf2 2023-03-19 09:54:23,668 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/c4346fea9abd482faa257d5314da5a09 2023-03-19 09:54:23,678 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/4675bcc8b4f247509ef959e5be87b6cf 2023-03-19 09:54:23,682 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/a4bf8d97226047199d56ee1df996bcf2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a4bf8d97226047199d56ee1df996bcf2 2023-03-19 09:54:23,686 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a4bf8d97226047199d56ee1df996bcf2, entries=150, sequenceid=142, filesize=25.3 K 2023-03-19 09:54:23,687 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/c4346fea9abd482faa257d5314da5a09 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/c4346fea9abd482faa257d5314da5a09 2023-03-19 09:54:23,690 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/c4346fea9abd482faa257d5314da5a09, entries=150, sequenceid=142, filesize=11.8 K 2023-03-19 09:54:23,691 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/4675bcc8b4f247509ef959e5be87b6cf as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/4675bcc8b4f247509ef959e5be87b6cf 2023-03-19 09:54:23,695 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/4675bcc8b4f247509ef959e5be87b6cf, entries=150, sequenceid=142, filesize=11.8 K 2023-03-19 09:54:23,696 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 4a5125924c57b3613b6b6094511223b4 in 863ms, sequenceid=142, compaction requested=true 2023-03-19 09:54:23,697 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:23,697 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:23,703 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@655c3e86[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:23,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:23,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,706 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:23,706 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,706 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:23,711 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:23,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219663076, max timestamp=1679219663711Num uniques 1077; 2023-03-19 09:54:23,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-19 09:54:23,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=107.34 KB heapSize=301.52 KB 2023-03-19 09:54:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:54:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:54:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:23,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:23,716 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:23,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:23,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:23,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:23,716 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:23,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:23,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,720 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:23,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:23,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:23,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,723 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:23,723 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:23,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:23,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:23,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:23,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:23,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:23,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219723769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,823 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:23,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:23,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:23,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:23,827 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:23,827 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:23,827 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:23,827 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:23,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:23,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:23,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:23,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:23,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:23,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:23,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:23,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:23,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:23,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f5a56b8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:23,838 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:23,838 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:23,840 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,840 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:23,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:23,843 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:23,843 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:23,843 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:23,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:23,844 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:23,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:23,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,850 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:23,850 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:23,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:23,850 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:23,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:23,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:23,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:23,850 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,851 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:23,851 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:23,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:23,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:23,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219723873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,951 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:23,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:23,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:23,955 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:23,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:23,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:23,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:23,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,956 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,957 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:23,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:23,957 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,958 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,958 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:23,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:23,959 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:23,959 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:23,959 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:23,960 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:23,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:23,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321a03e1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:23,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:23,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:23,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:23,970 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:23,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:23,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:23,972 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:23,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:23,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:23,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:23,972 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:23,974 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,974 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:23,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:23,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:23,979 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,981 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,981 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:23,981 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:23,981 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:23,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:23,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:23,981 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:23,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:23,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:23,982 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:23,982 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:23,981 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:23,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,983 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:23,983 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:24,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36092 deadline: 1679219724078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,082 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,087 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,087 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,087 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,087 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,089 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,091 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,091 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,091 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,091 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49595db[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,095 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,095 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,096 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,098 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,100 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,100 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,105 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,105 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,105 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,106 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,108 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:24,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:24,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:24,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,111 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:24,111 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:24,134 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319fbd00e4b859c41d8acc3a7b47a5bbb5d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319fbd00e4b859c41d8acc3a7b47a5bbb5d 2023-03-19 09:54:24,211 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,215 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,219 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,220 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b933c1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,225 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,227 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,230 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,231 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:24,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:24,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,234 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:24,234 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:24,334 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,338 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,341 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,342 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,343 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,345 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dab5af8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,350 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,350 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,351 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,353 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,353 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,358 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:24,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:24,361 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:24,361 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:24,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:24,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:24,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219724382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,461 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,464 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,465 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,465 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,465 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,465 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,466 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,468 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,468 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,468 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,468 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,470 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fcc5ba7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,477 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,482 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,482 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,484 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,484 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,484 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,485 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,486 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:24,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:24,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:24,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,492 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,492 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:24,492 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:24,492 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,544 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/c695cbda6fc84565bf21b377b5f7980e 2023-03-19 09:54:24,592 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,596 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,598 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,599 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,600 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,600 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,600 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,600 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219724599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,601 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,602 DEBUG [Thread-895] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,603 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@375f2ec8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,604 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,604 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,606 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219724604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,608 DEBUG [Thread-899] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:54:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,608 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219724609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,611 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,612 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:24,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219724610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,613 DEBUG [Thread-897] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:54:24,614 DEBUG [Thread-893] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:54:24,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,617 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:24,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:24,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:24,624 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:24,624 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:24,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,724 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,727 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,727 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,727 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,727 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,729 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,733 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e3124d9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,741 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,741 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,742 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,742 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,742 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:24,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:24,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,747 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:24,747 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:24,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:24,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,848 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,851 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,856 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,857 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,857 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@596add7d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,859 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,862 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,865 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,865 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,872 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:24,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:24,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,879 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:24,879 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:24,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219724892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/0621472284f6489facce05cf6c9af7af 2023-03-19 09:54:24,979 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:24,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:24,982 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:24,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:24,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:24,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,985 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:24,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:24,985 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:24,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:24,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:24,987 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,987 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:24,987 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:24,988 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:24,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2406b494[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:24,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:24,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,993 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:24,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:24,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:24,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:24,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:24,994 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:24,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:24,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:24,997 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:24,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:24,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:24,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:24,997 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:24,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:24,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:24,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,000 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:25,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:25,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:25,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:25,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,004 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,004 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,004 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,004 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,004 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:25,005 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:25,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:25,105 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:25,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:25,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:25,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:25,108 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:25,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:25,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:25,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:25,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:25,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:25,113 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,113 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:25,113 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:25,113 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:25,115 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:25,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a61e7b2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:25,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:25,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,119 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:25,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:25,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:25,123 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:25,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:25,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:25,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:25,124 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:25,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:25,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,133 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:25,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:25,134 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:25,134 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:25,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,234 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:25,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:25,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:25,237 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:25,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:25,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:25,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:25,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:25,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:25,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:25,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,249 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0101cb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:25,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:25,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,253 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:25,253 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:25,255 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:25,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:25,258 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:25,258 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:25,258 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:25,258 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:25,258 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:25,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:25,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,262 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,264 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:25,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:25,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:25,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:25,269 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:25,269 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:25,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:25,370 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:25,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:25,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:25,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:25,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:25,374 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:25,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:25,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:25,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,376 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,376 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c92e5dfc18c845368ffa46dd3a1ea4d2 2023-03-19 09:54:25,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:25,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:25,379 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,379 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:25,380 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,380 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:25,380 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:25,380 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:25,384 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/c695cbda6fc84565bf21b377b5f7980e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c695cbda6fc84565bf21b377b5f7980e 2023-03-19 09:54:25,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120fa623[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:25,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:25,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:25,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:25,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c695cbda6fc84565bf21b377b5f7980e, entries=200, sequenceid=197, filesize=32.1 K 2023-03-19 09:54:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:25,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/0621472284f6489facce05cf6c9af7af as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0621472284f6489facce05cf6c9af7af 2023-03-19 09:54:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:25,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:25,397 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:25,397 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:25,397 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:25,397 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:25,397 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:25,399 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,399 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:25,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,399 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:25,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,403 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,406 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:25,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:25,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:25,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:25,407 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,407 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:25,407 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:25,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:25,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0621472284f6489facce05cf6c9af7af, entries=150, sequenceid=197, filesize=11.8 K 2023-03-19 09:54:25,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c92e5dfc18c845368ffa46dd3a1ea4d2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c92e5dfc18c845368ffa46dd3a1ea4d2 2023-03-19 09:54:25,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c92e5dfc18c845368ffa46dd3a1ea4d2, entries=150, sequenceid=197, filesize=11.8 K 2023-03-19 09:54:25,419 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~301.47 KB/308704, currentSize=73.80 KB/75570 for 4a5125924c57b3613b6b6094511223b4 in 1705ms, sequenceid=197, compaction requested=true 2023-03-19 09:54:25,419 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:25,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:25,419 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-19 09:54:25,419 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-19 09:54:25,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:25,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:25,422 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 201538 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-19 09:54:25,423 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/A is initiating minor compaction (all files) 2023-03-19 09:54:25,423 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/A in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,423 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d8c1fbea1cf426a9ddffd408cfd66ee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dae7bbc9546d4812bfe9664a9211deee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d0f2728548749f3966200e4d3cead9f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/11921d427632417eaf32be2cb2c792c8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c55d3016429c4079bcda8236a65eda66, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a4bf8d97226047199d56ee1df996bcf2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c695cbda6fc84565bf21b377b5f7980e] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=196.8 K 2023-03-19 09:54:25,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81308 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-19 09:54:25,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/B is initiating minor compaction (all files) 2023-03-19 09:54:25,423 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/B in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,423 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/785cbbfae5e847778a14d8af7c722f46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/66ca74de0a964951a1e1bb998810654c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5a5e1b25eda14389abf66dfacc010029, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/725b7ae91a3d4e779b9924c037e04a3c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/019f7b4d7e61496dbb93b9fa35104a7b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/c4346fea9abd482faa257d5314da5a09, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0621472284f6489facce05cf6c9af7af] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=79.4 K 2023-03-19 09:54:25,424 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7d8c1fbea1cf426a9ddffd408cfd66ee, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679219657901 2023-03-19 09:54:25,424 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 785cbbfae5e847778a14d8af7c722f46, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679219657901 2023-03-19 09:54:25,425 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting dae7bbc9546d4812bfe9664a9211deee, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219658056 2023-03-19 09:54:25,425 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 66ca74de0a964951a1e1bb998810654c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219658056 2023-03-19 09:54:25,425 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7d0f2728548749f3966200e4d3cead9f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679219658199 2023-03-19 09:54:25,426 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5a5e1b25eda14389abf66dfacc010029, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679219658199 2023-03-19 09:54:25,426 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 11921d427632417eaf32be2cb2c792c8, keycount=300, bloomtype=ROW, size=45.5 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679219660359 2023-03-19 09:54:25,427 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 725b7ae91a3d4e779b9924c037e04a3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679219660360 2023-03-19 09:54:25,427 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c55d3016429c4079bcda8236a65eda66, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679219660464 2023-03-19 09:54:25,427 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 019f7b4d7e61496dbb93b9fa35104a7b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679219660464 2023-03-19 09:54:25,427 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a4bf8d97226047199d56ee1df996bcf2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679219662576 2023-03-19 09:54:25,428 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c4346fea9abd482faa257d5314da5a09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679219662576 2023-03-19 09:54:25,428 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c695cbda6fc84565bf21b377b5f7980e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219663076 2023-03-19 09:54:25,428 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0621472284f6489facce05cf6c9af7af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219663076 2023-03-19 09:54:25,455 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#B#compaction#137 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:25,476 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#A#compaction#136 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:25,484 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/79013e631a4d4fe58667b1e0d4930805 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/79013e631a4d4fe58667b1e0d4930805 2023-03-19 09:54:25,490 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/B of 4a5125924c57b3613b6b6094511223b4 into 79013e631a4d4fe58667b1e0d4930805(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:25,490 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:25,490 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/B, priority=9, startTime=1679219665419; duration=0sec 2023-03-19 09:54:25,490 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:25,490 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-19 09:54:25,493 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81308 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-19 09:54:25,493 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/C is initiating minor compaction (all files) 2023-03-19 09:54:25,493 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/C in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,493 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1f956f5e701e430181210f5e9c3d288a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/991efbaeb0d7413baf9c47a27f8196a0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/757eb84f76ec473ca0cc4d1bc75bceeb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/bf5705e07bb9410d9ba5b213293d6acd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/a447f8f51d81474f89c64b3878c20d49, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/4675bcc8b4f247509ef959e5be87b6cf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c92e5dfc18c845368ffa46dd3a1ea4d2] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=79.4 K 2023-03-19 09:54:25,494 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1f956f5e701e430181210f5e9c3d288a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679219657901 2023-03-19 09:54:25,495 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 991efbaeb0d7413baf9c47a27f8196a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219658056 2023-03-19 09:54:25,496 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 757eb84f76ec473ca0cc4d1bc75bceeb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679219658199 2023-03-19 09:54:25,496 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting bf5705e07bb9410d9ba5b213293d6acd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679219660360 2023-03-19 09:54:25,496 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a447f8f51d81474f89c64b3878c20d49, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679219660464 2023-03-19 09:54:25,497 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4675bcc8b4f247509ef959e5be87b6cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679219662576 2023-03-19 09:54:25,498 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c92e5dfc18c845368ffa46dd3a1ea4d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219663076 2023-03-19 09:54:25,507 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:25,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:25,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:25,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:25,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:25,521 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:25,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:25,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:25,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,523 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#C#compaction#138 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:25,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:25,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:25,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:25,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:25,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:25,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:25,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:25,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:25,540 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:25,541 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:25,541 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-19 09:54:25,542 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:25,542 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:25,542 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:25,542 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:25,542 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:25,542 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:25,549 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/9771a61c422c4c689c8bc7776f096cf3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9771a61c422c4c689c8bc7776f096cf3 2023-03-19 09:54:25,564 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/C of 4a5125924c57b3613b6b6094511223b4 into 9771a61c422c4c689c8bc7776f096cf3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:25,564 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:25,564 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/C, priority=9, startTime=1679219665419; duration=0sec 2023-03-19 09:54:25,565 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:25,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:25,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:25,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:25,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219725954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:25,992 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303196cb893df15934308820bfc4fcfff1384 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196cb893df15934308820bfc4fcfff1384 2023-03-19 09:54:25,997 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=212, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/1fba0474d612464f8de794e20e596580 2023-03-19 09:54:26,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:26,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219726056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:26,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219726264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,346 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/cd6777291dcb4cf59be83eddad8c9f7d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/cd6777291dcb4cf59be83eddad8c9f7d 2023-03-19 09:54:26,353 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/A of 4a5125924c57b3613b6b6094511223b4 into cd6777291dcb4cf59be83eddad8c9f7d(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:26,353 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:26,353 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/A, priority=9, startTime=1679219665419; duration=0sec 2023-03-19 09:54:26,354 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:26,414 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/191896cf0f584905ae2bd03d1195dd5b 2023-03-19 09:54:26,436 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/685bc87a7fe04030a7bfe88ce91d4083 2023-03-19 09:54:26,440 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/1fba0474d612464f8de794e20e596580 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/1fba0474d612464f8de794e20e596580 2023-03-19 09:54:26,444 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/1fba0474d612464f8de794e20e596580, entries=150, sequenceid=212, filesize=25.3 K 2023-03-19 09:54:26,445 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/191896cf0f584905ae2bd03d1195dd5b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/191896cf0f584905ae2bd03d1195dd5b 2023-03-19 09:54:26,449 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/191896cf0f584905ae2bd03d1195dd5b, entries=150, sequenceid=212, filesize=11.8 K 2023-03-19 09:54:26,450 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/685bc87a7fe04030a7bfe88ce91d4083 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/685bc87a7fe04030a7bfe88ce91d4083 2023-03-19 09:54:26,454 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/685bc87a7fe04030a7bfe88ce91d4083, entries=150, sequenceid=212, filesize=11.8 K 2023-03-19 09:54:26,455 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4a5125924c57b3613b6b6094511223b4 in 914ms, sequenceid=212, compaction requested=false 2023-03-19 09:54:26,455 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:26,455 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:26,456 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:26,457 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,458 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,458 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:26,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c12e2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:26,460 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:26,460 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:26,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:26,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:26,461 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,461 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:26,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:26,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:26,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:26,465 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:26,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:26,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:26,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:26,466 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:26,469 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,469 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,469 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,469 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:26,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:26,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,474 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:26,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,474 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:26,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:26,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-19 09:54:26,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:26,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:26,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:26,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:26,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:26,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:26,575 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:26,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:26,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:26,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:26,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:26,580 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:26,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:26,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:26,581 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,581 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,582 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:26,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:26,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:26,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:26,586 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:26,586 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,586 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:26,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:26,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:26,609 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,611 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66852940[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:26,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:26,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:26,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219726608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,616 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:26,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:26,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,620 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319e91dd66c57ea4831a290c1f4f9f29229 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e91dd66c57ea4831a290c1f4f9f29229 2023-03-19 09:54:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:26,622 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:26,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:26,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:26,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:26,622 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:26,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,629 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,629 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:26,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:26,629 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,629 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,629 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:26,629 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:26,629 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,629 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,634 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=233, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/ec483e4c22464486b3091f0a1d9ed252 2023-03-19 09:54:26,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/1f3075b6c01e4d12bb80bd8b6e5616b9 2023-03-19 09:54:26,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219726716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,730 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:26,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:26,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:26,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:26,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:26,732 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:26,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:26,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:26,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,735 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:26,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:26,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:26,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:26,737 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,737 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:26,737 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:26,737 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:26,739 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:26,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731f7302[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:26,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:26,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,745 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:26,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:26,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:26,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:26,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:26,751 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:26,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:26,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:26,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:26,751 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:26,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,756 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:26,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:26,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,759 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,759 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,759 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,759 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,759 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:26,759 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:26,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,859 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:26,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:26,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:26,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:26,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:26,867 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:26,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:26,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:26,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,869 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,870 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,870 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:26,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:26,871 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,871 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:26,871 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:26,871 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:26,873 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,873 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c4eedfc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:26,875 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:26,875 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:26,877 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,877 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:26,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:26,881 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:26,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:26,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:26,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:26,881 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:26,882 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,882 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,883 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,884 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:26,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,888 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:26,888 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,888 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:26,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:26,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219726921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,988 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:26,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:26,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:26,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:26,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:26,992 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:26,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:26,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:26,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,992 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,994 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:26,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:26,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:26,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:26,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:26,995 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,995 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:26,995 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:26,995 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,997 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f865795[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:26,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:26,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:27,000 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:27,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:27,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:27,001 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:27,001 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:27,001 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:27,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:27,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:27,006 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:27,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:27,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:27,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:27,006 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:27,007 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,007 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:27,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:27,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:27,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:27,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:27,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:27,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:27,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:27,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,012 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:27,013 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:27,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:27,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/8c6a83a8812b49b7a5d82ca307a5a69c 2023-03-19 09:54:27,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/ec483e4c22464486b3091f0a1d9ed252 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ec483e4c22464486b3091f0a1d9ed252 2023-03-19 09:54:27,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ec483e4c22464486b3091f0a1d9ed252, entries=200, sequenceid=233, filesize=32.1 K 2023-03-19 09:54:27,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/1f3075b6c01e4d12bb80bd8b6e5616b9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1f3075b6c01e4d12bb80bd8b6e5616b9 2023-03-19 09:54:27,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1f3075b6c01e4d12bb80bd8b6e5616b9, entries=150, sequenceid=233, filesize=11.8 K 2023-03-19 09:54:27,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/8c6a83a8812b49b7a5d82ca307a5a69c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/8c6a83a8812b49b7a5d82ca307a5a69c 2023-03-19 09:54:27,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/8c6a83a8812b49b7a5d82ca307a5a69c, entries=150, sequenceid=233, filesize=11.8 K 2023-03-19 09:54:27,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4a5125924c57b3613b6b6094511223b4 in 515ms, sequenceid=233, compaction requested=true 2023-03-19 09:54:27,086 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:27,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:27,086 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:27,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:27,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:27,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:27,088 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:27,088 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/A is initiating minor compaction (all files) 2023-03-19 09:54:27,088 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/A in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:27,088 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:27,088 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/cd6777291dcb4cf59be83eddad8c9f7d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/1fba0474d612464f8de794e20e596580, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ec483e4c22464486b3091f0a1d9ed252] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=82.9 K 2023-03-19 09:54:27,088 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/B is initiating minor compaction (all files) 2023-03-19 09:54:27,088 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/B in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:27,088 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/79013e631a4d4fe58667b1e0d4930805, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/191896cf0f584905ae2bd03d1195dd5b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1f3075b6c01e4d12bb80bd8b6e5616b9] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=35.5 K 2023-03-19 09:54:27,088 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting cd6777291dcb4cf59be83eddad8c9f7d, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219663076 2023-03-19 09:54:27,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 79013e631a4d4fe58667b1e0d4930805, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219663076 2023-03-19 09:54:27,089 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1fba0474d612464f8de794e20e596580, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679219663760 2023-03-19 09:54:27,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 191896cf0f584905ae2bd03d1195dd5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679219663760 2023-03-19 09:54:27,090 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ec483e4c22464486b3091f0a1d9ed252, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679219665946 2023-03-19 09:54:27,090 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1f3075b6c01e4d12bb80bd8b6e5616b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679219665946 2023-03-19 09:54:27,101 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#B#compaction#145 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:27,113 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:27,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:27,120 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:27,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:27,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:27,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:27,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:27,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:27,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:27,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:27,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:27,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:27,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:27,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:27,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:27,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:27,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:27,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:27,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:27,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:27,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:27,129 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:54:27,130 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#A#compaction#146 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:27,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:27,144 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/5ee3bd00935940fe8fe54b3ff24d61e6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5ee3bd00935940fe8fe54b3ff24d61e6 2023-03-19 09:54:27,159 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/B of 4a5125924c57b3613b6b6094511223b4 into 5ee3bd00935940fe8fe54b3ff24d61e6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:27,159 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:27,159 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/B, priority=13, startTime=1679219667086; duration=0sec 2023-03-19 09:54:27,160 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:27,160 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:27,163 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:27,163 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/C is initiating minor compaction (all files) 2023-03-19 09:54:27,163 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/C in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:27,163 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9771a61c422c4c689c8bc7776f096cf3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/685bc87a7fe04030a7bfe88ce91d4083, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/8c6a83a8812b49b7a5d82ca307a5a69c] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=35.5 K 2023-03-19 09:54:27,164 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9771a61c422c4c689c8bc7776f096cf3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679219663076 2023-03-19 09:54:27,165 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 685bc87a7fe04030a7bfe88ce91d4083, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679219663760 2023-03-19 09:54:27,165 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8c6a83a8812b49b7a5d82ca307a5a69c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679219665946 2023-03-19 09:54:27,178 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#C#compaction#148 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:27,197 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319e1de6627bb0a4459bcf8ba0ff2c2e6ad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e1de6627bb0a4459bcf8ba0ff2c2e6ad 2023-03-19 09:54:27,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:27,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:27,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:27,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219727284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:27,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219727389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:27,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219727591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:27,629 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/46ca05b78dce4d95afbad8686fee420a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/46ca05b78dce4d95afbad8686fee420a 2023-03-19 09:54:27,630 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/05737612ecc4427cbbf5bef106f19af9 2023-03-19 09:54:27,637 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/C of 4a5125924c57b3613b6b6094511223b4 into 46ca05b78dce4d95afbad8686fee420a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:27,637 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:27,637 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/C, priority=13, startTime=1679219667087; duration=0sec 2023-03-19 09:54:27,637 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:27,649 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/dd4afc4c172a47039ad2ae33c393c939 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dd4afc4c172a47039ad2ae33c393c939 2023-03-19 09:54:27,656 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/A of 4a5125924c57b3613b6b6094511223b4 into dd4afc4c172a47039ad2ae33c393c939(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:27,656 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:27,656 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/A, priority=13, startTime=1679219667086; duration=0sec 2023-03-19 09:54:27,656 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:27,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:27,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219727894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,062 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/8cbb74a56ff34af89ff5b992f9825f2b 2023-03-19 09:54:28,074 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/cc806c2c3a3742c18b222a1db245fd26 2023-03-19 09:54:28,078 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/05737612ecc4427cbbf5bef106f19af9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/05737612ecc4427cbbf5bef106f19af9 2023-03-19 09:54:28,082 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/05737612ecc4427cbbf5bef106f19af9, entries=150, sequenceid=246, filesize=25.3 K 2023-03-19 09:54:28,083 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/8cbb74a56ff34af89ff5b992f9825f2b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/8cbb74a56ff34af89ff5b992f9825f2b 2023-03-19 09:54:28,088 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/8cbb74a56ff34af89ff5b992f9825f2b, entries=150, sequenceid=246, filesize=11.8 K 2023-03-19 09:54:28,088 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/cc806c2c3a3742c18b222a1db245fd26 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/cc806c2c3a3742c18b222a1db245fd26 2023-03-19 09:54:28,092 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/cc806c2c3a3742c18b222a1db245fd26, entries=150, sequenceid=246, filesize=11.8 K 2023-03-19 09:54:28,093 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4a5125924c57b3613b6b6094511223b4 in 964ms, sequenceid=246, compaction requested=false 2023-03-19 09:54:28,093 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:28,093 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:28,096 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3006ae0c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:28,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:28,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,099 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,099 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:28,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:28,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:28,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:28,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:28,103 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:28,103 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:28,103 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:28,103 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:28,103 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:28,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:28,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,113 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,113 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:28,113 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:28,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,214 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:28,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:28,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:28,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:28,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:28,217 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:28,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:28,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:28,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,219 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,219 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,219 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:28,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:28,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:28,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:28,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:28,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:28,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:28,221 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:28,248 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031981a2629ed88048a1a477ffa4512f0d44 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031981a2629ed88048a1a477ffa4512f0d44 2023-03-19 09:54:28,259 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/d49f66e32e6549f5900fde4bd4956497 2023-03-19 09:54:28,279 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/539056418c194d3eb219c1930a7157f5 2023-03-19 09:54:28,326 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c4236c280c6f4bb2ae8569a5355ce232 2023-03-19 09:54:28,334 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/d49f66e32e6549f5900fde4bd4956497 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d49f66e32e6549f5900fde4bd4956497 2023-03-19 09:54:28,340 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d49f66e32e6549f5900fde4bd4956497, entries=150, sequenceid=268, filesize=25.4 K 2023-03-19 09:54:28,341 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/539056418c194d3eb219c1930a7157f5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/539056418c194d3eb219c1930a7157f5 2023-03-19 09:54:28,346 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/539056418c194d3eb219c1930a7157f5, entries=150, sequenceid=268, filesize=11.9 K 2023-03-19 09:54:28,346 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c4236c280c6f4bb2ae8569a5355ce232 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c4236c280c6f4bb2ae8569a5355ce232 2023-03-19 09:54:28,352 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c4236c280c6f4bb2ae8569a5355ce232, entries=150, sequenceid=268, filesize=11.9 K 2023-03-19 09:54:28,353 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 4a5125924c57b3613b6b6094511223b4 in 132ms, sequenceid=268, compaction requested=true 2023-03-19 09:54:28,353 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:28,353 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:28,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:28,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:28,355 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,355 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38895c6d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:28,359 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:28,359 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,361 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,362 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:28,362 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:28,362 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:28,362 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,362 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:28,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,364 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:28,364 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,364 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:28,367 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:28,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:28,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:28,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:28,367 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:28,369 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,369 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,369 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:28,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:28,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:28,375 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:28,375 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:28,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,375 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,375 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,475 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:28,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:28,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:28,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:28,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:28,479 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:28,479 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,479 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:28,479 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:28,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:28,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:28,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:28,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:28,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:28,496 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:28,496 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:28,496 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,496 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:28,497 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:28,497 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:28,499 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,499 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,499 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5343678d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:28,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:28,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,502 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:28,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:28,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:28,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:28,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:28,507 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:28,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:28,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:28,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:28,508 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:28,511 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,511 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,511 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,514 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:28,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:28,523 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:28,523 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:28,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,523 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,523 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,523 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:28,571 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:54:28,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:28,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:28,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:28,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:28,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:28,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:28,612 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303195029377d145d4d9db16148d34eae0c2d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195029377d145d4d9db16148d34eae0c2d 2023-03-19 09:54:28,623 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:28,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:28,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219668620, max timestamp=1679219668624Num uniques 859; 2023-03-19 09:54:28,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219668620, max timestamp=1679219668624Num uniques 859; 2023-03-19 09:54:28,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:28,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:28,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219668620, max timestamp=1679219668624Num uniques 859; 2023-03-19 09:54:28,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:28,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:28,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:28,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:28,629 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:28,629 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,629 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:28,629 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:28,630 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,630 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,631 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,631 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,631 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:28,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:28,633 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,633 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:28,633 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:28,633 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:28,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,634 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:28,634 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:28,635 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,635 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,636 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,636 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:28,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:28,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb4a88f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:28,638 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:28,638 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,640 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:28,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:28,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:28,640 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,640 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:28,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:28,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:28,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:28,644 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:28,645 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:28,645 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:28,645 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:28,645 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:28,646 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,647 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,648 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,655 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:28,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:28,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:28,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,655 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:28,655 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:28,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219728659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219728659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219728660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219728661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219728662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,755 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:28,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:28,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:28,759 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:28,759 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,759 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:28,759 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:28,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:28,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:28,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:28,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:28,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:28,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:28,763 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:28,763 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,764 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:28,764 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:28,764 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:28,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,765 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219728763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:28,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219728763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219728764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:28,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219728767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b47039b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:28,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219728767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,770 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,770 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,770 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:28,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:28,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:28,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:28,772 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:28,772 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,772 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:28,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:28,775 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:28,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:28,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:28,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:28,775 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:28,776 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,776 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:28,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:28,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:28,780 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,780 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:28,781 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:28,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,881 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:28,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:28,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:28,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:28,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:28,885 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:28,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:28,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:28,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,887 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:28,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:28,889 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,889 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:28,889 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:28,889 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:28,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,892 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69df5eaa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:28,893 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:28,893 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:28,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:28,896 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:28,899 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:28,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:28,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:28,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:28,899 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:28,900 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,900 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,900 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,901 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:28,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:28,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,905 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:28,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:28,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:28,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:28,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:28,906 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:28,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:28,906 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:28,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:28,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:28,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:28,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:28,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:28,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:28,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219728968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219728968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219728968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219728970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:28,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:28,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219728972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,006 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:29,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:29,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:29,009 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:29,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:29,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:29,010 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,010 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,011 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:29,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:29,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:29,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:29,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:29,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:29,013 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,013 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:29,014 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:29,014 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:29,016 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21e83313[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:29,019 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:29,019 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,020 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:29,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:29,022 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/ba1042dfd3264f548720475b21bd27af 2023-03-19 09:54:29,025 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:29,034 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:29,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:29,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:29,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:29,034 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:29,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,038 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,039 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:29,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,045 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,046 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,046 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:29,046 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:29,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,146 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:29,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:29,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:29,149 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:29,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:29,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:29,150 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,150 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,152 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:29,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:29,153 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,154 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:29,154 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:29,154 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:29,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:29,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:29,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,155 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a230771[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:29,157 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:29,157 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:29,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:29,160 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:29,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:29,163 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:29,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:29,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:29,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:29,163 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:29,165 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,165 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,165 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,166 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,167 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,169 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,169 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:29,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:29,169 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,169 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,169 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,170 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,170 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:29,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,170 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:29,270 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:29,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:29,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219729275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219729275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:29,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:29,278 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:29,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:29,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:29,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219729276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219729277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219729277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,280 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,280 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:29,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:29,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:29,288 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,288 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:29,288 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:29,288 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:29,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,290 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,290 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,290 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:29,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395b9475[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:29,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:29,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,295 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,295 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:29,295 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:29,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:29,295 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:29,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:29,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:29,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:29,299 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:29,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:29,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:29,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:29,299 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:29,301 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,301 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,301 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,303 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,306 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:29,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,306 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,306 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:29,306 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:29,406 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:29,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:29,410 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:29,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:29,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:29,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,412 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,413 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,413 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:29,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:29,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:29,415 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,415 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:29,415 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:29,415 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:29,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:29,416 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:29,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a62a7fa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:29,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:29,420 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,420 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:29,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:29,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:29,422 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:29,424 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:29,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:29,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:29,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:29,424 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:29,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:29,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:29,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:29,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,432 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:29,432 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:29,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:29,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:29,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:29,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/99a22dc37ffc41c48049c41259b0f7ec 2023-03-19 09:54:29,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c04251f9793e4e8387a640da00878296 2023-03-19 09:54:29,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/ba1042dfd3264f548720475b21bd27af as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ba1042dfd3264f548720475b21bd27af 2023-03-19 09:54:29,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ba1042dfd3264f548720475b21bd27af, entries=250, sequenceid=279, filesize=39.2 K 2023-03-19 09:54:29,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/99a22dc37ffc41c48049c41259b0f7ec as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/99a22dc37ffc41c48049c41259b0f7ec 2023-03-19 09:54:29,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/99a22dc37ffc41c48049c41259b0f7ec, entries=150, sequenceid=279, filesize=11.9 K 2023-03-19 09:54:29,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c04251f9793e4e8387a640da00878296 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c04251f9793e4e8387a640da00878296 2023-03-19 09:54:29,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c04251f9793e4e8387a640da00878296, entries=150, sequenceid=279, filesize=11.9 K 2023-03-19 09:54:29,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4a5125924c57b3613b6b6094511223b4 in 906ms, sequenceid=279, compaction requested=true 2023-03-19 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:29,477 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:29,477 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:29,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:29,479 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48736 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:29,479 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118260 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:29,479 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/B is initiating minor compaction (all files) 2023-03-19 09:54:29,479 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/A is initiating minor compaction (all files) 2023-03-19 09:54:29,479 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/B in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,479 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/A in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,479 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5ee3bd00935940fe8fe54b3ff24d61e6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/8cbb74a56ff34af89ff5b992f9825f2b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/539056418c194d3eb219c1930a7157f5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/99a22dc37ffc41c48049c41259b0f7ec] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=47.6 K 2023-03-19 09:54:29,479 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dd4afc4c172a47039ad2ae33c393c939, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/05737612ecc4427cbbf5bef106f19af9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d49f66e32e6549f5900fde4bd4956497, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ba1042dfd3264f548720475b21bd27af] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=115.5 K 2023-03-19 09:54:29,480 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5ee3bd00935940fe8fe54b3ff24d61e6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679219665946 2023-03-19 09:54:29,480 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting dd4afc4c172a47039ad2ae33c393c939, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679219665946 2023-03-19 09:54:29,480 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8cbb74a56ff34af89ff5b992f9825f2b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1679219666600 2023-03-19 09:54:29,480 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 05737612ecc4427cbbf5bef106f19af9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1679219666600 2023-03-19 09:54:29,481 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 539056418c194d3eb219c1930a7157f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219667235 2023-03-19 09:54:29,481 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d49f66e32e6549f5900fde4bd4956497, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219667235 2023-03-19 09:54:29,481 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 99a22dc37ffc41c48049c41259b0f7ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679219668506 2023-03-19 09:54:29,481 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ba1042dfd3264f548720475b21bd27af, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679219668506 2023-03-19 09:54:29,491 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#B#compaction#157 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:29,502 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#A#compaction#158 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:29,511 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/fbe5678d29f64b7e8b406822c34f44a3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fbe5678d29f64b7e8b406822c34f44a3 2023-03-19 09:54:29,517 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/B of 4a5125924c57b3613b6b6094511223b4 into fbe5678d29f64b7e8b406822c34f44a3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:29,517 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:29,517 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/B, priority=12, startTime=1679219669477; duration=0sec 2023-03-19 09:54:29,518 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:29,518 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:29,519 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48736 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:29,519 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/C is initiating minor compaction (all files) 2023-03-19 09:54:29,519 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/C in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,520 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/46ca05b78dce4d95afbad8686fee420a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/cc806c2c3a3742c18b222a1db245fd26, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c4236c280c6f4bb2ae8569a5355ce232, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c04251f9793e4e8387a640da00878296] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=47.6 K 2023-03-19 09:54:29,520 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 46ca05b78dce4d95afbad8686fee420a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679219665946 2023-03-19 09:54:29,520 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting cc806c2c3a3742c18b222a1db245fd26, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1679219666600 2023-03-19 09:54:29,521 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c4236c280c6f4bb2ae8569a5355ce232, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679219667235 2023-03-19 09:54:29,521 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c04251f9793e4e8387a640da00878296, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679219668506 2023-03-19 09:54:29,533 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:29,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:29,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:29,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:29,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:29,537 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:29,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:29,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:29,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,537 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:29,539 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#C#compaction#159 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:29,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:29,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:29,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:29,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:29,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:29,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:29,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:29,548 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:29,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:29,579 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/f741b9b944944c36b6e5500f73ba5134 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/f741b9b944944c36b6e5500f73ba5134 2023-03-19 09:54:29,585 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/C of 4a5125924c57b3613b6b6094511223b4 into f741b9b944944c36b6e5500f73ba5134(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:29,585 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:29,585 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/C, priority=12, startTime=1679219669477; duration=0sec 2023-03-19 09:54:29,585 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:29,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:29,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:29,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219729785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219729786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219729786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219729787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219729787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219729888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219729888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219729890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219729893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219729893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:29,994 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303198567e233938143c7b3c82840cf2219b3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198567e233938143c7b3c82840cf2219b3 2023-03-19 09:54:30,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219730092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219730092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219730092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219730097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219730099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219730396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219730396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219730397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,400 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/7e07cdd931da4ca1939ea2bc62bedcac 2023-03-19 09:54:30,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219730399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219730401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,741 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/d0a75252294e408fa9a2b759c5bdf85f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d0a75252294e408fa9a2b759c5bdf85f 2023-03-19 09:54:30,748 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/A of 4a5125924c57b3613b6b6094511223b4 into d0a75252294e408fa9a2b759c5bdf85f(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-19 09:54:30,749 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:30,749 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/A, priority=12, startTime=1679219669477; duration=1sec 2023-03-19 09:54:30,749 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:30,812 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/6c84cdf1b8bd46a3be048e29c0fe0420 2023-03-19 09:54:30,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219730898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219730899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219730901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219730906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:30,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:30,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219730907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,224 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/304ad18cf7fa4746805f0a631d4e7666 2023-03-19 09:54:31,229 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/7e07cdd931da4ca1939ea2bc62bedcac as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7e07cdd931da4ca1939ea2bc62bedcac 2023-03-19 09:54:31,233 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7e07cdd931da4ca1939ea2bc62bedcac, entries=150, sequenceid=317, filesize=25.4 K 2023-03-19 09:54:31,234 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/6c84cdf1b8bd46a3be048e29c0fe0420 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/6c84cdf1b8bd46a3be048e29c0fe0420 2023-03-19 09:54:31,238 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/6c84cdf1b8bd46a3be048e29c0fe0420, entries=150, sequenceid=317, filesize=11.9 K 2023-03-19 09:54:31,238 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/304ad18cf7fa4746805f0a631d4e7666 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/304ad18cf7fa4746805f0a631d4e7666 2023-03-19 09:54:31,242 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/304ad18cf7fa4746805f0a631d4e7666, entries=150, sequenceid=317, filesize=11.9 K 2023-03-19 09:54:31,243 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 4a5125924c57b3613b6b6094511223b4 in 1695ms, sequenceid=317, compaction requested=false 2023-03-19 09:54:31,243 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:31,243 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:31,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:31,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:31,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:31,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:31,249 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:31,249 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:31,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:31,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:31,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:31,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af2690e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:31,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:31,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,253 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:31,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:31,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:31,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:31,273 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:31,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:31,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:31,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:31,273 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:31,274 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,275 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:31,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,275 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:31,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:31,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:31,282 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:31,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:31,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:31,283 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:31,283 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:31,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:31,383 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:31,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:31,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:31,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:31,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:31,389 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:31,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:31,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:31,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:31,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:31,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:31,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:31,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:31,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:31,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:31,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:31,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:31,394 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:31,394 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:31,439 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319e4420c264b9b4e0595b26408739f6932 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e4420c264b9b4e0595b26408739f6932 2023-03-19 09:54:31,450 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/5c79e967ef054242845b251151f1779a 2023-03-19 09:54:31,466 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/fb0aeb7b5c8f4d7e92f62f7b405358e8 2023-03-19 09:54:31,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:31,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:31,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679219671916, max timestamp=1679219671926Num uniques 859; 2023-03-19 09:54:31,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-19 09:54:31,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679219671916, max timestamp=1679219671926Num uniques 859; 2023-03-19 09:54:31,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679219671916, max timestamp=1679219671926Num uniques 859; 2023-03-19 09:54:31,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-19 09:54:31,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-19 09:54:31,934 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/b32709c3001f4868b63db3e3f82a5594 2023-03-19 09:54:31,939 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/5c79e967ef054242845b251151f1779a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/5c79e967ef054242845b251151f1779a 2023-03-19 09:54:31,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:31,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219731937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:31,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219731939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:31,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219731939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:31,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219731940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:31,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219731940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,943 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/5c79e967ef054242845b251151f1779a, entries=150, sequenceid=329, filesize=25.4 K 2023-03-19 09:54:31,945 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/fb0aeb7b5c8f4d7e92f62f7b405358e8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fb0aeb7b5c8f4d7e92f62f7b405358e8 2023-03-19 09:54:31,949 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fb0aeb7b5c8f4d7e92f62f7b405358e8, entries=150, sequenceid=329, filesize=11.9 K 2023-03-19 09:54:31,950 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/b32709c3001f4868b63db3e3f82a5594 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/b32709c3001f4868b63db3e3f82a5594 2023-03-19 09:54:31,954 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/b32709c3001f4868b63db3e3f82a5594, entries=150, sequenceid=329, filesize=11.9 K 2023-03-19 09:54:31,955 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 4a5125924c57b3613b6b6094511223b4 in 561ms, sequenceid=329, compaction requested=true 2023-03-19 09:54:31,955 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:31,955 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:31,955 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:31,955 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:31,955 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,955 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:31,955 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:31,957 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:31,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4868f7ae[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:31,961 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:31,961 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:31,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:31,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:31,965 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:31,965 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:31,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,965 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:31,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:31,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:31,965 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:31,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:31,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:31,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:31,965 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:31,966 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,966 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:31,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:31,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:31,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:31,970 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:31,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:31,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:31,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:31,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:31,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:31,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:31,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:31,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:31,971 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:31,971 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:31,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:31,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:32,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-19 09:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:54:32,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219732045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219732049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219732050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219732051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219732051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,071 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,079 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,086 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2df52aea[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,095 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,095 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,095 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,095 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:32,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,098 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,098 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,101 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:32,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:32,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:32,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,108 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:32,108 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:32,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219732152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219732153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219732154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219732155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219732155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,208 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,212 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,214 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121715d9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,229 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,229 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,229 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,230 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,230 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:32,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,234 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,234 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:32,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:32,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,241 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:32,241 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,341 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,345 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,349 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,349 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,349 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,349 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,349 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,351 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b74b48c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,356 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,356 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219732354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219732355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,357 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219732356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219732358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219732358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,360 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,360 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:32,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:32,367 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:32,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,368 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,368 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:32,459 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303199eaa43df96a842499c579e08093f2bbb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303199eaa43df96a842499c579e08093f2bbb 2023-03-19 09:54:32,465 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/52543ee8de104e8dada41f127043c2a4 2023-03-19 09:54:32,468 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,472 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,473 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,476 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,476 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,476 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,476 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,478 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,479 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442effcb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,479 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,483 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,483 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,483 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,483 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:32,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,486 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,486 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,486 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,486 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,486 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,488 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,488 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,488 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:32,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:32,494 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,494 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,494 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,494 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,494 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,494 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,494 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:32,494 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:32,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,594 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,598 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,601 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,601 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,601 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,602 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,602 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,602 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,602 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,603 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,603 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40882fde[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,606 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,612 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,612 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:32,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:32,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:32,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,619 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:32,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,619 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:32,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219732658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219732658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219732661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219732661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219732662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,719 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,722 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,727 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,727 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,727 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,730 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9b43cb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,733 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:32,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,737 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,737 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,740 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:32,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:32,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,744 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:32,744 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:32,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,745 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,745 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,844 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,848 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,850 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,852 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,852 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,852 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,852 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,854 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4255ac0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,857 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,857 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:32,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,864 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,864 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,864 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,864 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,864 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,870 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,870 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:32,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:32,870 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,871 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:32,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,871 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:32,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:32,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/df9a6aeea2094c92baaa21ca4c281071 2023-03-19 09:54:32,971 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:32,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:32,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:32,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:32,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:32,974 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:32,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:32,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:32,975 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,975 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:32,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:32,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:32,978 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:32,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,982 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,983 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,983 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c63d28[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:32,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:32,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:32,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:32,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:32,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:32,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:32,987 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:32,989 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:32,989 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:32,989 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:32,989 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:32,989 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:32,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:32,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:32,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:32,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:33,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:33,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:33,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,008 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:33,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:33,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:33,008 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:33,109 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:33,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:33,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:33,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:33,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:33,112 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:33,112 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:33,112 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:33,112 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:33,113 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,113 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,114 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:33,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:33,114 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,114 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:33,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:33,117 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,117 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:33,117 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:33,117 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:33,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:33,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:33,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:33,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:33,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:33,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:33,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:33,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@363102d1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:33,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:33,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,125 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:33,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:33,125 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:33,125 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:33,128 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:33,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:33,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:33,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:33,131 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:33,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:33,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:33,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:33,131 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:33,132 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,132 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,132 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:33,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:33,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:33,133 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:33,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:33,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:33,137 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,137 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:33,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:33,137 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:33,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:33,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:33,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219733161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:33,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219733162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:33,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219733164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:33,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219733168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219733169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,237 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:33,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:33,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:33,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:33,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:33,240 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:33,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:33,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:33,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:33,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,242 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:33,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:33,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:33,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:33,244 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,244 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:33,244 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:33,244 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:33,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:33,247 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:33,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:33,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6848ac7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:33,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:33,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:33,251 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,251 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:33,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:33,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:33,252 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:33,254 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:33,255 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:33,255 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:33,255 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:33,255 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:33,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:33,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:33,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:33,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,257 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:33,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:33,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:33,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:33,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:33,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:33,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:33,261 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:33,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:33,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:33,262 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:33,262 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:33,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:33,262 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/e34c74493c7144e5a3f12ce71f1c2214 2023-03-19 09:54:33,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/52543ee8de104e8dada41f127043c2a4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/52543ee8de104e8dada41f127043c2a4 2023-03-19 09:54:33,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/52543ee8de104e8dada41f127043c2a4, entries=300, sequenceid=367, filesize=46.1 K 2023-03-19 09:54:33,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/df9a6aeea2094c92baaa21ca4c281071 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/df9a6aeea2094c92baaa21ca4c281071 2023-03-19 09:54:33,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/df9a6aeea2094c92baaa21ca4c281071, entries=150, sequenceid=367, filesize=11.9 K 2023-03-19 09:54:33,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/e34c74493c7144e5a3f12ce71f1c2214 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/e34c74493c7144e5a3f12ce71f1c2214 2023-03-19 09:54:33,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/e34c74493c7144e5a3f12ce71f1c2214, entries=150, sequenceid=367, filesize=11.9 K 2023-03-19 09:54:33,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~451.27 KB/462096, currentSize=26.84 KB/27480 for 4a5125924c57b3613b6b6094511223b4 in 1280ms, sequenceid=367, compaction requested=true 2023-03-19 09:54:33,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:33,325 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:33,325 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:33,325 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:33,325 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:33,325 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:33,346 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125786 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:33,347 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/A is initiating minor compaction (all files) 2023-03-19 09:54:33,347 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/A in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,347 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d0a75252294e408fa9a2b759c5bdf85f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7e07cdd931da4ca1939ea2bc62bedcac, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/5c79e967ef054242845b251151f1779a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/52543ee8de104e8dada41f127043c2a4] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=122.8 K 2023-03-19 09:54:33,347 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d0a75252294e408fa9a2b759c5bdf85f, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679219668506 2023-03-19 09:54:33,348 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7e07cdd931da4ca1939ea2bc62bedcac, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679219668659 2023-03-19 09:54:33,348 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:33,348 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/B is initiating minor compaction (all files) 2023-03-19 09:54:33,348 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5c79e967ef054242845b251151f1779a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1679219669784 2023-03-19 09:54:33,348 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/B in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,348 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fbe5678d29f64b7e8b406822c34f44a3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/6c84cdf1b8bd46a3be048e29c0fe0420, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fb0aeb7b5c8f4d7e92f62f7b405358e8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/df9a6aeea2094c92baaa21ca4c281071] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=48.1 K 2023-03-19 09:54:33,349 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 52543ee8de104e8dada41f127043c2a4, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679219671938 2023-03-19 09:54:33,349 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fbe5678d29f64b7e8b406822c34f44a3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679219668506 2023-03-19 09:54:33,349 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6c84cdf1b8bd46a3be048e29c0fe0420, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679219668659 2023-03-19 09:54:33,350 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fb0aeb7b5c8f4d7e92f62f7b405358e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1679219669784 2023-03-19 09:54:33,351 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting df9a6aeea2094c92baaa21ca4c281071, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679219672043 2023-03-19 09:54:33,362 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:33,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:33,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:33,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:33,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:33,369 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#B#compaction#170 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:33,370 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#A#compaction#169 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:33,371 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:33,371 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:33,371 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:33,371 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:33,373 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,373 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:33,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:33,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:33,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:33,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:33,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:33,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:33,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:33,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:33,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:33,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:33,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:33,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:33,400 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-19 09:54:33,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:33,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:33,401 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:33,401 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:33,401 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:33,401 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:33,410 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/051217659bc645c6a0114d0eef37d62e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/051217659bc645c6a0114d0eef37d62e 2023-03-19 09:54:33,416 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/B of 4a5125924c57b3613b6b6094511223b4 into 051217659bc645c6a0114d0eef37d62e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:33,416 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:33,416 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/B, priority=12, startTime=1679219673325; duration=0sec 2023-03-19 09:54:33,416 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:33,416 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:33,418 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:33,418 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/C is initiating minor compaction (all files) 2023-03-19 09:54:33,418 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/C in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:33,418 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/f741b9b944944c36b6e5500f73ba5134, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/304ad18cf7fa4746805f0a631d4e7666, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/b32709c3001f4868b63db3e3f82a5594, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/e34c74493c7144e5a3f12ce71f1c2214] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=48.1 K 2023-03-19 09:54:33,418 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f741b9b944944c36b6e5500f73ba5134, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679219668506 2023-03-19 09:54:33,419 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 304ad18cf7fa4746805f0a631d4e7666, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679219668659 2023-03-19 09:54:33,419 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b32709c3001f4868b63db3e3f82a5594, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1679219669784 2023-03-19 09:54:33,420 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e34c74493c7144e5a3f12ce71f1c2214, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679219672043 2023-03-19 09:54:33,428 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#C#compaction#172 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:33,822 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031957b9898be2234290a2ea9eaa778270a6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031957b9898be2234290a2ea9eaa778270a6 2023-03-19 09:54:33,838 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/0721f969cb7e49e99f7482a9e397cdf6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/0721f969cb7e49e99f7482a9e397cdf6 2023-03-19 09:54:33,845 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/C of 4a5125924c57b3613b6b6094511223b4 into 0721f969cb7e49e99f7482a9e397cdf6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:33,845 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:33,845 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/C, priority=12, startTime=1679219673325; duration=0sec 2023-03-19 09:54:33,845 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:34,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:34,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:34,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679219674179, max timestamp=1679219674182Num uniques 859; 2023-03-19 09:54:34,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679219674179, max timestamp=1679219674182Num uniques 859; 2023-03-19 09:54:34,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:54:34,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:54:34,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679219674179, max timestamp=1679219674182Num uniques 859; 2023-03-19 09:54:34,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:54:34,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219734195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219734196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219734197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219734197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679219674192, max timestamp=1679219674199Num uniques 1118; 2023-03-19 09:54:34,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-03-19 09:54:34,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679219674192, max timestamp=1679219674199Num uniques 1118; 2023-03-19 09:54:34,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679219674192, max timestamp=1679219674199Num uniques 1118; 2023-03-19 09:54:34,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-03-19 09:54:34,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-03-19 09:54:34,233 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/b26efa0210554b0094b341b6393fef13 2023-03-19 09:54:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219674250, max timestamp=1679219674267Num uniques 1177; 2023-03-19 09:54:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-19 09:54:34,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219674250, max timestamp=1679219674267Num uniques 1177; 2023-03-19 09:54:34,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-19 09:54:34,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219674250, max timestamp=1679219674267Num uniques 1177; 2023-03-19 09:54:34,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-19 09:54:34,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1679219674294, max timestamp=1679219674309Num uniques 1036; 2023-03-19 09:54:34,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-03-19 09:54:34,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1679219674294, max timestamp=1679219674309Num uniques 1036; 2023-03-19 09:54:34,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-03-19 09:54:34,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1679219674294, max timestamp=1679219674309Num uniques 1036; 2023-03-19 09:54:34,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-03-19 09:54:34,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219734321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219734322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219734322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219734325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219734325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219734429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219734429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219734429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219734429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219734429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,623 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/ae8bd3e86ed7430791385096f8f0a0f0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ae8bd3e86ed7430791385096f8f0a0f0 2023-03-19 09:54:34,629 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/A of 4a5125924c57b3613b6b6094511223b4 into ae8bd3e86ed7430791385096f8f0a0f0(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-19 09:54:34,629 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:34,629 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/A, priority=12, startTime=1679219673325; duration=1sec 2023-03-19 09:54:34,629 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:34,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219734632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219734632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219734633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219734633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:34,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219734634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,650 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/0570e9d2279545139d240a3277e2bb94 2023-03-19 09:54:34,663 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/ce1823b1d68745b38e05dbd3ab948cf8 2023-03-19 09:54:34,668 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/b26efa0210554b0094b341b6393fef13 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b26efa0210554b0094b341b6393fef13 2023-03-19 09:54:34,672 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b26efa0210554b0094b341b6393fef13, entries=150, sequenceid=374, filesize=25.4 K 2023-03-19 09:54:34,673 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/0570e9d2279545139d240a3277e2bb94 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0570e9d2279545139d240a3277e2bb94 2023-03-19 09:54:34,677 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0570e9d2279545139d240a3277e2bb94, entries=150, sequenceid=374, filesize=11.9 K 2023-03-19 09:54:34,678 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/ce1823b1d68745b38e05dbd3ab948cf8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ce1823b1d68745b38e05dbd3ab948cf8 2023-03-19 09:54:34,682 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ce1823b1d68745b38e05dbd3ab948cf8, entries=150, sequenceid=374, filesize=11.9 K 2023-03-19 09:54:34,682 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 4a5125924c57b3613b6b6094511223b4 in 1282ms, sequenceid=374, compaction requested=false 2023-03-19 09:54:34,682 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:34,682 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:34,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:34,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:34,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:34,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:34,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:34,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:34,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:34,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d9338d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:34,687 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:34,687 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:34,689 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:34,689 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:34,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:34,691 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:34,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:34,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:34,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:34,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:34,693 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:34,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:34,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:34,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:34,694 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:34,695 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,695 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:34,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,695 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:34,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:34,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,697 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:34,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:34,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:34,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:34,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:34,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:34,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:34,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:34,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:34,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:34,702 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:34,703 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:34,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,703 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,703 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,703 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,803 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:34,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:34,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:34,806 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:34,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:34,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:34,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:34,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:34,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:34,808 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:34,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:34,809 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,809 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:34,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:34,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:34,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:34,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:34,810 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:34,810 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:34,810 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:34,810 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:34,810 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:34,810 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:34,810 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:34,810 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=161.02 KB heapSize=443.39 KB 2023-03-19 09:54:34,810 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:34,810 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:34,810 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:34,810 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:34,811 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:34,811 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:34,829 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319b03c7205540a4a21b63711ab01cc5832 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b03c7205540a4a21b63711ab01cc5832 2023-03-19 09:54:34,835 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/bc6e7f03627649c1802cbafcfb6e50fe 2023-03-19 09:54:34,845 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/dfa04aacd7fc4c47ab0b7985fda79cea 2023-03-19 09:54:34,857 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/d18f16b9a9614c788608741d93b9876e 2023-03-19 09:54:34,860 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/bc6e7f03627649c1802cbafcfb6e50fe as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/bc6e7f03627649c1802cbafcfb6e50fe 2023-03-19 09:54:34,865 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/bc6e7f03627649c1802cbafcfb6e50fe, entries=150, sequenceid=465, filesize=25.4 K 2023-03-19 09:54:34,866 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/dfa04aacd7fc4c47ab0b7985fda79cea as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/dfa04aacd7fc4c47ab0b7985fda79cea 2023-03-19 09:54:34,871 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/dfa04aacd7fc4c47ab0b7985fda79cea, entries=150, sequenceid=465, filesize=11.9 K 2023-03-19 09:54:34,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/d18f16b9a9614c788608741d93b9876e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d18f16b9a9614c788608741d93b9876e 2023-03-19 09:54:34,877 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d18f16b9a9614c788608741d93b9876e, entries=150, sequenceid=465, filesize=11.9 K 2023-03-19 09:54:34,878 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~443.34 KB/453984, currentSize=0 B/0 for 4a5125924c57b3613b6b6094511223b4 in 68ms, sequenceid=465, compaction requested=true 2023-03-19 09:54:34,878 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:34,878 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:34,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:34,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:34,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:34,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e7064c3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:34,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:34,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,892 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:34,892 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,893 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:34,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:34,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:34,897 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:34,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:34,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:34,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:34,897 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:34,900 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,900 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:34,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,900 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:34,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:34,900 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:34,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:34,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:34,905 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:34,905 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:34,905 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:34,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:34,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:34,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:34,970 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:54:34,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:34,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:34,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:34,971 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:34,971 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:34,971 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:34,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:34,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679219674971, max timestamp=1679219674997Num uniques 859; 2023-03-19 09:54:34,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679219674971, max timestamp=1679219674997Num uniques 859; 2023-03-19 09:54:34,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-19 09:54:34,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-19 09:54:34,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679219674971, max timestamp=1679219674997Num uniques 859; 2023-03-19 09:54:34,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-19 09:54:35,005 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,010 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,010 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,010 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,010 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219735004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219735006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219735007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219735005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219735007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,019 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,019 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,019 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:35,019 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:35,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,021 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:35,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1069dc56[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:35,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:35,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,026 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,026 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:35,026 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:35,026 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:35,026 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,026 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:35,027 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:35,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:35,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:35,030 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:35,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:35,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:35,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:35,030 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:35,032 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,032 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,032 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:35,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:35,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,037 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:35,037 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:35,037 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36148 deadline: 1679219735111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219735112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219735112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219735112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219735112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,138 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,141 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,143 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,145 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,145 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,145 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:35,146 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:35,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:35,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:35,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:35,147 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:35,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:35,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3219ae2e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:35,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,150 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:35,150 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,150 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:35,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:35,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:35,152 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:35,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:35,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:35,155 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:35,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:35,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:35,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:35,155 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:35,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,160 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:35,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,163 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:35,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:35,163 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:35,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,263 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,267 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,271 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,273 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,273 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,273 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:35,273 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:35,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:35,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:35,276 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53bdb6bb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:35,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:35,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,279 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:35,280 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,280 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:35,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:35,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:35,284 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:35,284 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:35,284 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:35,284 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:35,284 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:35,286 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,286 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,286 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,286 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:35,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:35,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,292 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:35,292 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:35,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219735314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219735314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36142 deadline: 1679219735314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219735315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219735316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,392 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,395 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,396 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,396 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,398 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,399 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,400 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031962d43d95240d4308b130b104f1d31b43 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031962d43d95240d4308b130b104f1d31b43 2023-03-19 09:54:35,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:35,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:35,402 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:35,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec68673[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:35,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:35,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,408 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:35,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:35,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:35,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:35,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:35,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:35,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:35,413 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=475, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/b0a4cccbfee74f9cbee3171662edabb4 2023-03-19 09:54:35,412 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:35,413 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:35,413 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:35,413 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:35,413 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:35,415 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,415 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,415 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,417 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:35,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:35,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,420 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:35,420 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:35,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/1976e42b44644a53b3580d1cda37152a 2023-03-19 09:54:35,520 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,523 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,529 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:35,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:35,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:35,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:35,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20937e91[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:35,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:35,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,534 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:35,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:35,534 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,535 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:35,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:35,536 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,536 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,536 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,536 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:35,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:35,538 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:35,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:35,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:35,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:35,538 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:35,541 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,541 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,541 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,543 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,550 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,550 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:35,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,552 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:35,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:35,553 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:35,553 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:35,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219735617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219735617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219735618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219735618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:35,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219735618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,653 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,656 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,658 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:35,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:35,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:35,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:35,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:35,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:35,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f04cc73[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:35,663 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:35,663 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,669 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:35,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:35,670 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:35,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:35,673 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:35,673 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:35,673 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:35,673 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:35,673 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:35,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,676 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:35,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:35,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,680 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:35,680 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,681 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,780 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,784 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,788 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,788 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,788 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:35,788 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:35,790 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,790 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6132b146[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:35,792 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:35,792 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,794 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:35,795 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,795 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:35,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,797 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:35,797 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,797 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:35,799 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:35,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:35,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:35,800 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:35,800 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:35,801 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,801 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,801 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:35,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:35,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:35,807 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:35,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:35,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:35,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:35,808 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:35,808 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:35,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/7109159d383044c4bc3c4d63fda8c333 2023-03-19 09:54:35,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/b0a4cccbfee74f9cbee3171662edabb4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b0a4cccbfee74f9cbee3171662edabb4 2023-03-19 09:54:35,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b0a4cccbfee74f9cbee3171662edabb4, entries=350, sequenceid=475, filesize=52.9 K 2023-03-19 09:54:35,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/1976e42b44644a53b3580d1cda37152a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1976e42b44644a53b3580d1cda37152a 2023-03-19 09:54:35,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1976e42b44644a53b3580d1cda37152a, entries=150, sequenceid=475, filesize=11.9 K 2023-03-19 09:54:35,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/7109159d383044c4bc3c4d63fda8c333 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/7109159d383044c4bc3c4d63fda8c333 2023-03-19 09:54:35,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/7109159d383044c4bc3c4d63fda8c333, entries=150, sequenceid=475, filesize=11.9 K 2023-03-19 09:54:35,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=161.02 KB/164880 for 4a5125924c57b3613b6b6094511223b4 in 896ms, sequenceid=475, compaction requested=true 2023-03-19 09:54:35,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:35,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:35,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:35,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:35,866 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:35,866 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:35,868 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 132966 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:35,868 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/A is initiating minor compaction (all files) 2023-03-19 09:54:35,868 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:35,868 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/A in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,868 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/B is initiating minor compaction (all files) 2023-03-19 09:54:35,868 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ae8bd3e86ed7430791385096f8f0a0f0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b26efa0210554b0094b341b6393fef13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/bc6e7f03627649c1802cbafcfb6e50fe, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b0a4cccbfee74f9cbee3171662edabb4] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=129.8 K 2023-03-19 09:54:35,868 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/B in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,868 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/051217659bc645c6a0114d0eef37d62e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0570e9d2279545139d240a3277e2bb94, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/dfa04aacd7fc4c47ab0b7985fda79cea, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1976e42b44644a53b3580d1cda37152a] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=48.2 K 2023-03-19 09:54:35,869 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ae8bd3e86ed7430791385096f8f0a0f0, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679219672043 2023-03-19 09:54:35,869 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 051217659bc645c6a0114d0eef37d62e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679219672043 2023-03-19 09:54:35,869 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b26efa0210554b0094b341b6393fef13, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679219672048 2023-03-19 09:54:35,869 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0570e9d2279545139d240a3277e2bb94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679219672048 2023-03-19 09:54:35,870 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting bc6e7f03627649c1802cbafcfb6e50fe, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219674321 2023-03-19 09:54:35,870 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting dfa04aacd7fc4c47ab0b7985fda79cea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219674321 2023-03-19 09:54:35,870 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b0a4cccbfee74f9cbee3171662edabb4, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1679219674959 2023-03-19 09:54:35,870 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1976e42b44644a53b3580d1cda37152a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1679219674962 2023-03-19 09:54:35,883 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#B#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:35,897 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#A#compaction#181 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:35,908 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:35,909 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/39e1105102b54cc7a30ca36dd52e7d93 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/39e1105102b54cc7a30ca36dd52e7d93 2023-03-19 09:54:35,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:35,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:35,915 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/B of 4a5125924c57b3613b6b6094511223b4 into 39e1105102b54cc7a30ca36dd52e7d93(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:35,915 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:35,915 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/B, priority=12, startTime=1679219675866; duration=0sec 2023-03-19 09:54:35,915 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:35,915 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:35,917 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:35,918 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 4a5125924c57b3613b6b6094511223b4/C is initiating minor compaction (all files) 2023-03-19 09:54:35,918 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4a5125924c57b3613b6b6094511223b4/C in TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,918 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/0721f969cb7e49e99f7482a9e397cdf6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ce1823b1d68745b38e05dbd3ab948cf8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d18f16b9a9614c788608741d93b9876e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/7109159d383044c4bc3c4d63fda8c333] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp, totalSize=48.2 K 2023-03-19 09:54:35,918 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0721f969cb7e49e99f7482a9e397cdf6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679219672043 2023-03-19 09:54:35,918 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ce1823b1d68745b38e05dbd3ab948cf8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679219672048 2023-03-19 09:54:35,919 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d18f16b9a9614c788608741d93b9876e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219674321 2023-03-19 09:54:35,919 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 7109159d383044c4bc3c4d63fda8c333, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1679219674962 2023-03-19 09:54:35,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:35,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:35,921 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:35,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:35,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:35,922 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,922 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,924 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,924 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:35,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:35,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:35,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:35,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:35,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:35,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:35,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:35,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:35,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:35,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:35,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:35,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:35,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:35,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:35,926 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:35,926 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:35,926 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=161.02 KB heapSize=399.05 KB 2023-03-19 09:54:35,926 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:35,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:35,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:35,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:35,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:35,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:35,928 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 4a5125924c57b3613b6b6094511223b4#C#compaction#183 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:36,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:36,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:36,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1679219736128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36094 deadline: 1679219736128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219736131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219736133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219736134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219736235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219736235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219736238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,347 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/9796172bc0924015b73f636b70e05bc0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9796172bc0924015b73f636b70e05bc0 2023-03-19 09:54:36,359 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303197aaca9dc26674813a8ee2fca72108a6b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197aaca9dc26674813a8ee2fca72108a6b 2023-03-19 09:54:36,384 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/C of 4a5125924c57b3613b6b6094511223b4 into 9796172bc0924015b73f636b70e05bc0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:36,384 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:36,384 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/C, priority=12, startTime=1679219675866; duration=0sec 2023-03-19 09:54:36,384 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:36,391 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=503, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/a47dfe5efada47baa8b5ecf69ecd26b5 2023-03-19 09:54:36,426 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/11459215d8484047ba85307d895e8434 2023-03-19 09:54:36,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219736440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219736440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219736442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219736749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219736753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:36,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219736753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,757 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/a513c1eebeb44fe08e7224ce74796419 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a513c1eebeb44fe08e7224ce74796419 2023-03-19 09:54:36,762 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4a5125924c57b3613b6b6094511223b4/A of 4a5125924c57b3613b6b6094511223b4 into a513c1eebeb44fe08e7224ce74796419(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:36,762 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:36,762 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4., storeName=4a5125924c57b3613b6b6094511223b4/A, priority=12, startTime=1679219675866; duration=0sec 2023-03-19 09:54:36,762 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:36,855 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/d0d9dde991f14fdf9961100fb3f78595 2023-03-19 09:54:36,860 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/a47dfe5efada47baa8b5ecf69ecd26b5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a47dfe5efada47baa8b5ecf69ecd26b5 2023-03-19 09:54:36,864 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a47dfe5efada47baa8b5ecf69ecd26b5, entries=150, sequenceid=503, filesize=25.4 K 2023-03-19 09:54:36,865 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/11459215d8484047ba85307d895e8434 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/11459215d8484047ba85307d895e8434 2023-03-19 09:54:36,870 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/11459215d8484047ba85307d895e8434, entries=150, sequenceid=503, filesize=11.9 K 2023-03-19 09:54:36,870 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/d0d9dde991f14fdf9961100fb3f78595 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d0d9dde991f14fdf9961100fb3f78595 2023-03-19 09:54:36,874 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d0d9dde991f14fdf9961100fb3f78595, entries=150, sequenceid=503, filesize=11.9 K 2023-03-19 09:54:36,875 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~399 KB/408576, currentSize=40.25 KB/41220 for 4a5125924c57b3613b6b6094511223b4 in 949ms, sequenceid=503, compaction requested=false 2023-03-19 09:54:36,875 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:36,875 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:36,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:36,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:36,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:36,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:36,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:36,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:36,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:36,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0d460d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:36,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:36,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:36,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:36,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:36,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:36,883 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:36,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:36,886 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:36,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:36,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:36,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:36,886 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:36,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:36,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:36,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:36,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:36,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:36,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,893 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:36,893 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:36,893 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:36,894 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:36,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:36,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:36,894 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:36,894 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:36,893 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:36,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:36,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:36,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:36,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:36,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:36,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:36,994 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:36,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:36,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:36,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:36,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:36,998 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:36,998 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:36,998 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:36,998 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:36,998 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:36,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:37,000 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:37,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:37,000 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:37,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:37,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:37,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:37,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:37,002 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:37,002 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. started... 2023-03-19 09:54:37,002 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-19 09:54:37,002 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:37,002 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:37,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:37,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. as already flushing 2023-03-19 09:54:37,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219677203, max timestamp=1679219677211Num uniques 859; 2023-03-19 09:54:37,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219677203, max timestamp=1679219677211Num uniques 859; 2023-03-19 09:54:37,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:37,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219677203, max timestamp=1679219677211Num uniques 859; 2023-03-19 09:54:37,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:37,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:54:37,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36094 deadline: 1679219737247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219677229, max timestamp=1679219677250Num uniques 1018; 2023-03-19 09:54:37,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:54:37,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219677229, max timestamp=1679219677250Num uniques 1018; 2023-03-19 09:54:37,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:54:37,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219677229, max timestamp=1679219677250Num uniques 1018; 2023-03-19 09:54:37,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:54:37,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219737276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219737306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1679219737306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219737308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36094 deadline: 1679219737353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219737409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1679219737412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219737412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219737412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,439 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319b3b6e461bf8847abb4979fd95e755522 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b3b6e461bf8847abb4979fd95e755522 2023-03-19 09:54:37,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36094 deadline: 1679219737558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219737616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219737617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1679219737617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219737618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,858 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=514, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/96aea353dca4434d94aba765aaaf2989 2023-03-19 09:54:37,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36094 deadline: 1679219737863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,867 DEBUG [Thread-902] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:54245 2023-03-19 09:54:37,867 DEBUG [Thread-902] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:37,868 DEBUG [Thread-904] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:54245 2023-03-19 09:54:37,868 DEBUG [Thread-904] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:37,869 DEBUG [Thread-906] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:54245 2023-03-19 09:54:37,869 DEBUG [Thread-906] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:37,874 DEBUG [Thread-908] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58e27a32 to 127.0.0.1:54245 2023-03-19 09:54:37,874 DEBUG [Thread-908] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:37,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1679219737919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219737919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219737920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:37,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219737922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,271 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/a2d3114c49094b1f8322a5566edef869 2023-03-19 09:54:38,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:38,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36094 deadline: 1679219738368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:38,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1679219738420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:38,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36148 deadline: 1679219738420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:38,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36092 deadline: 1679219738421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:38,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36142 deadline: 1679219738425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,685 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/ae15ce1642f7454e945fbe8e91567124 2023-03-19 09:54:38,689 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/96aea353dca4434d94aba765aaaf2989 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/96aea353dca4434d94aba765aaaf2989 2023-03-19 09:54:38,692 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/96aea353dca4434d94aba765aaaf2989, entries=100, sequenceid=514, filesize=18.6 K 2023-03-19 09:54:38,693 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/a2d3114c49094b1f8322a5566edef869 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/a2d3114c49094b1f8322a5566edef869 2023-03-19 09:54:38,696 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/a2d3114c49094b1f8322a5566edef869, entries=100, sequenceid=514, filesize=9.5 K 2023-03-19 09:54:38,697 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/ae15ce1642f7454e945fbe8e91567124 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ae15ce1642f7454e945fbe8e91567124 2023-03-19 09:54:38,700 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ae15ce1642f7454e945fbe8e91567124, entries=100, sequenceid=514, filesize=9.5 K 2023-03-19 09:54:38,700 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 4a5125924c57b3613b6b6094511223b4 in 1698ms, sequenceid=514, compaction requested=true 2023-03-19 09:54:38,701 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:38,701 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:38,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:38,702 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:38,703 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,703 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:38,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:38,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d1d20b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:38,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:38,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:38,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:38,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:38,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:38,710 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:38,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:38,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:38,712 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:38,712 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:38,712 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:38,712 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:38,712 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:38,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:38,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:38,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:38,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:38,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:38,714 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:38,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:38,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:38,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:38,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:38,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:38,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:38,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:38,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:38,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:38,718 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:38,718 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:38,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:38,719 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,719 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:38,719 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:38,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:38,819 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:38,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:39,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:39,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-03-19 09:54:39,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:39,378 DEBUG [Thread-893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:54245 2023-03-19 09:54:39,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:39,378 DEBUG [Thread-893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:39,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:39,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:39,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:39,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:39,426 DEBUG [Thread-891] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x051c62b6 to 127.0.0.1:54245 2023-03-19 09:54:39,426 DEBUG [Thread-891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:39,428 DEBUG [Thread-899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:54245 2023-03-19 09:54:39,428 DEBUG [Thread-899] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:39,430 DEBUG [Thread-895] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21265fcb to 127.0.0.1:54245 2023-03-19 09:54:39,430 DEBUG [Thread-895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:39,430 DEBUG [Thread-897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:54245 2023-03-19 09:54:39,431 DEBUG [Thread-897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 221 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 4836 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 4557 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1999 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5996 rows 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1998 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5993 rows 2023-03-19 09:54:39,431 INFO [Listener at localhost/45961] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-19 09:54:39,431 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b8d0eb2 to 127.0.0.1:54245 2023-03-19 09:54:39,431 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:54:39,437 INFO [Listener at localhost/45961] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-19 09:54:39,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-19 09:54:39,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:39,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-19 09:54:39,441 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219679440"}]},"ts":"1679219679440"} 2023-03-19 09:54:39,441 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-19 09:54:39,535 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-19 09:54:39,536 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, UNASSIGN}] 2023-03-19 09:54:39,537 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, UNASSIGN 2023-03-19 09:54:39,538 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:39,538 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219679538"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219679538"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219679538"}]},"ts":"1679219679538"} 2023-03-19 09:54:39,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:54:39,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-19 09:54:39,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:39,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4a5125924c57b3613b6b6094511223b4, disabling compactions & flushes 2023-03-19 09:54:39,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:39,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-19 09:54:39,790 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031900138bf324f44784bb180a40449a99b0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031900138bf324f44784bb180a40449a99b0 2023-03-19 09:54:39,875 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-19 09:54:40,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-19 09:54:40,195 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=568, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/c3733108b1104f838209fe3f5d22bba7 2023-03-19 09:54:40,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-19 09:54:40,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/05ba8c78642f487fb473925dd77fc219 2023-03-19 09:54:41,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/1720613b475440048e7b5d376ceebe52 2023-03-19 09:54:41,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/c3733108b1104f838209fe3f5d22bba7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c3733108b1104f838209fe3f5d22bba7 2023-03-19 09:54:41,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c3733108b1104f838209fe3f5d22bba7, entries=150, sequenceid=568, filesize=25.4 K 2023-03-19 09:54:41,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/05ba8c78642f487fb473925dd77fc219 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/05ba8c78642f487fb473925dd77fc219 2023-03-19 09:54:41,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/05ba8c78642f487fb473925dd77fc219, entries=150, sequenceid=568, filesize=11.9 K 2023-03-19 09:54:41,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/1720613b475440048e7b5d376ceebe52 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1720613b475440048e7b5d376ceebe52 2023-03-19 09:54:41,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1720613b475440048e7b5d376ceebe52, entries=150, sequenceid=568, filesize=11.9 K 2023-03-19 09:54:41,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=26.84 KB/27480 for 4a5125924c57b3613b6b6094511223b4 in 1667ms, sequenceid=568, compaction requested=true 2023-03-19 09:54:41,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:41,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:41,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:41,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:41,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:41,045 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. because compaction request was cancelled 2023-03-19 09:54:41,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:41,045 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. because compaction request was cancelled 2023-03-19 09:54:41,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. after waiting 0 ms 2023-03-19 09:54:41,046 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. because compaction request was cancelled 2023-03-19 09:54:41,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:41,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4a5125924c57b3613b6b6094511223b4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-19 09:54:41,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=A 2023-03-19 09:54:41,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:41,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=B 2023-03-19 09:54:41,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:41,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4a5125924c57b3613b6b6094511223b4, store=C 2023-03-19 09:54:41,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:41,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303198aaadcefd51a4cc4b21c69d850ebb116 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198aaadcefd51a4cc4b21c69d850ebb116 2023-03-19 09:54:41,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-19 09:54:41,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=575, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/87c8b3e1c4c746d1938967300058b676 2023-03-19 09:54:42,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/eb3ae3ab68fb4f688bfbd3f08e942026 2023-03-19 09:54:42,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c7c8e5acc1574f13bb9d7f000fa7c892 2023-03-19 09:54:42,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/A/87c8b3e1c4c746d1938967300058b676 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/87c8b3e1c4c746d1938967300058b676 2023-03-19 09:54:42,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/87c8b3e1c4c746d1938967300058b676, entries=100, sequenceid=575, filesize=18.6 K 2023-03-19 09:54:42,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/B/eb3ae3ab68fb4f688bfbd3f08e942026 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/eb3ae3ab68fb4f688bfbd3f08e942026 2023-03-19 09:54:42,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/eb3ae3ab68fb4f688bfbd3f08e942026, entries=100, sequenceid=575, filesize=9.5 K 2023-03-19 09:54:42,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/.tmp/C/c7c8e5acc1574f13bb9d7f000fa7c892 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c7c8e5acc1574f13bb9d7f000fa7c892 2023-03-19 09:54:42,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c7c8e5acc1574f13bb9d7f000fa7c892, entries=100, sequenceid=575, filesize=9.5 K 2023-03-19 09:54:42,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 4a5125924c57b3613b6b6094511223b4 in 1654ms, sequenceid=575, compaction requested=true 2023-03-19 09:54:42,718 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d8c1fbea1cf426a9ddffd408cfd66ee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dae7bbc9546d4812bfe9664a9211deee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d0f2728548749f3966200e4d3cead9f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/11921d427632417eaf32be2cb2c792c8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c55d3016429c4079bcda8236a65eda66, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a4bf8d97226047199d56ee1df996bcf2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c695cbda6fc84565bf21b377b5f7980e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/cd6777291dcb4cf59be83eddad8c9f7d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/1fba0474d612464f8de794e20e596580, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ec483e4c22464486b3091f0a1d9ed252, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dd4afc4c172a47039ad2ae33c393c939, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/05737612ecc4427cbbf5bef106f19af9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d49f66e32e6549f5900fde4bd4956497, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ba1042dfd3264f548720475b21bd27af, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d0a75252294e408fa9a2b759c5bdf85f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7e07cdd931da4ca1939ea2bc62bedcac, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/5c79e967ef054242845b251151f1779a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/52543ee8de104e8dada41f127043c2a4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ae8bd3e86ed7430791385096f8f0a0f0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b26efa0210554b0094b341b6393fef13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/bc6e7f03627649c1802cbafcfb6e50fe, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b0a4cccbfee74f9cbee3171662edabb4] to archive 2023-03-19 09:54:42,719 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:54:42,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d8c1fbea1cf426a9ddffd408cfd66ee to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d8c1fbea1cf426a9ddffd408cfd66ee 2023-03-19 09:54:42,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dae7bbc9546d4812bfe9664a9211deee to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dae7bbc9546d4812bfe9664a9211deee 2023-03-19 09:54:42,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d0f2728548749f3966200e4d3cead9f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7d0f2728548749f3966200e4d3cead9f 2023-03-19 09:54:42,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/11921d427632417eaf32be2cb2c792c8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/11921d427632417eaf32be2cb2c792c8 2023-03-19 09:54:42,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c55d3016429c4079bcda8236a65eda66 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c55d3016429c4079bcda8236a65eda66 2023-03-19 09:54:42,725 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a4bf8d97226047199d56ee1df996bcf2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a4bf8d97226047199d56ee1df996bcf2 2023-03-19 09:54:42,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c695cbda6fc84565bf21b377b5f7980e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c695cbda6fc84565bf21b377b5f7980e 2023-03-19 09:54:42,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/cd6777291dcb4cf59be83eddad8c9f7d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/cd6777291dcb4cf59be83eddad8c9f7d 2023-03-19 09:54:42,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/1fba0474d612464f8de794e20e596580 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/1fba0474d612464f8de794e20e596580 2023-03-19 09:54:42,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ec483e4c22464486b3091f0a1d9ed252 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ec483e4c22464486b3091f0a1d9ed252 2023-03-19 09:54:42,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dd4afc4c172a47039ad2ae33c393c939 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/dd4afc4c172a47039ad2ae33c393c939 2023-03-19 09:54:42,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/05737612ecc4427cbbf5bef106f19af9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/05737612ecc4427cbbf5bef106f19af9 2023-03-19 09:54:42,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d49f66e32e6549f5900fde4bd4956497 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d49f66e32e6549f5900fde4bd4956497 2023-03-19 09:54:42,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ba1042dfd3264f548720475b21bd27af to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ba1042dfd3264f548720475b21bd27af 2023-03-19 09:54:42,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d0a75252294e408fa9a2b759c5bdf85f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/d0a75252294e408fa9a2b759c5bdf85f 2023-03-19 09:54:42,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7e07cdd931da4ca1939ea2bc62bedcac to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/7e07cdd931da4ca1939ea2bc62bedcac 2023-03-19 09:54:42,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/5c79e967ef054242845b251151f1779a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/5c79e967ef054242845b251151f1779a 2023-03-19 09:54:42,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/52543ee8de104e8dada41f127043c2a4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/52543ee8de104e8dada41f127043c2a4 2023-03-19 09:54:42,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ae8bd3e86ed7430791385096f8f0a0f0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/ae8bd3e86ed7430791385096f8f0a0f0 2023-03-19 09:54:42,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b26efa0210554b0094b341b6393fef13 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b26efa0210554b0094b341b6393fef13 2023-03-19 09:54:42,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/bc6e7f03627649c1802cbafcfb6e50fe to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/bc6e7f03627649c1802cbafcfb6e50fe 2023-03-19 09:54:42,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b0a4cccbfee74f9cbee3171662edabb4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/b0a4cccbfee74f9cbee3171662edabb4 2023-03-19 09:54:42,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/785cbbfae5e847778a14d8af7c722f46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/66ca74de0a964951a1e1bb998810654c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5a5e1b25eda14389abf66dfacc010029, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/725b7ae91a3d4e779b9924c037e04a3c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/019f7b4d7e61496dbb93b9fa35104a7b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/c4346fea9abd482faa257d5314da5a09, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/79013e631a4d4fe58667b1e0d4930805, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0621472284f6489facce05cf6c9af7af, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/191896cf0f584905ae2bd03d1195dd5b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5ee3bd00935940fe8fe54b3ff24d61e6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1f3075b6c01e4d12bb80bd8b6e5616b9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/8cbb74a56ff34af89ff5b992f9825f2b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/539056418c194d3eb219c1930a7157f5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fbe5678d29f64b7e8b406822c34f44a3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/99a22dc37ffc41c48049c41259b0f7ec, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/6c84cdf1b8bd46a3be048e29c0fe0420, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fb0aeb7b5c8f4d7e92f62f7b405358e8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/051217659bc645c6a0114d0eef37d62e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/df9a6aeea2094c92baaa21ca4c281071, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0570e9d2279545139d240a3277e2bb94, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/dfa04aacd7fc4c47ab0b7985fda79cea, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1976e42b44644a53b3580d1cda37152a] to archive 2023-03-19 09:54:42,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:54:42,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/785cbbfae5e847778a14d8af7c722f46 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/785cbbfae5e847778a14d8af7c722f46 2023-03-19 09:54:42,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/66ca74de0a964951a1e1bb998810654c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/66ca74de0a964951a1e1bb998810654c 2023-03-19 09:54:42,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5a5e1b25eda14389abf66dfacc010029 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5a5e1b25eda14389abf66dfacc010029 2023-03-19 09:54:42,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/725b7ae91a3d4e779b9924c037e04a3c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/725b7ae91a3d4e779b9924c037e04a3c 2023-03-19 09:54:42,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/019f7b4d7e61496dbb93b9fa35104a7b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/019f7b4d7e61496dbb93b9fa35104a7b 2023-03-19 09:54:42,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/c4346fea9abd482faa257d5314da5a09 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/c4346fea9abd482faa257d5314da5a09 2023-03-19 09:54:42,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/79013e631a4d4fe58667b1e0d4930805 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/79013e631a4d4fe58667b1e0d4930805 2023-03-19 09:54:42,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0621472284f6489facce05cf6c9af7af to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0621472284f6489facce05cf6c9af7af 2023-03-19 09:54:42,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/191896cf0f584905ae2bd03d1195dd5b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/191896cf0f584905ae2bd03d1195dd5b 2023-03-19 09:54:42,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5ee3bd00935940fe8fe54b3ff24d61e6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/5ee3bd00935940fe8fe54b3ff24d61e6 2023-03-19 09:54:42,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1f3075b6c01e4d12bb80bd8b6e5616b9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1f3075b6c01e4d12bb80bd8b6e5616b9 2023-03-19 09:54:42,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/8cbb74a56ff34af89ff5b992f9825f2b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/8cbb74a56ff34af89ff5b992f9825f2b 2023-03-19 09:54:42,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/539056418c194d3eb219c1930a7157f5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/539056418c194d3eb219c1930a7157f5 2023-03-19 09:54:42,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fbe5678d29f64b7e8b406822c34f44a3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fbe5678d29f64b7e8b406822c34f44a3 2023-03-19 09:54:42,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/99a22dc37ffc41c48049c41259b0f7ec to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/99a22dc37ffc41c48049c41259b0f7ec 2023-03-19 09:54:42,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/6c84cdf1b8bd46a3be048e29c0fe0420 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/6c84cdf1b8bd46a3be048e29c0fe0420 2023-03-19 09:54:42,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fb0aeb7b5c8f4d7e92f62f7b405358e8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/fb0aeb7b5c8f4d7e92f62f7b405358e8 2023-03-19 09:54:42,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/051217659bc645c6a0114d0eef37d62e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/051217659bc645c6a0114d0eef37d62e 2023-03-19 09:54:42,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/df9a6aeea2094c92baaa21ca4c281071 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/df9a6aeea2094c92baaa21ca4c281071 2023-03-19 09:54:42,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0570e9d2279545139d240a3277e2bb94 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/0570e9d2279545139d240a3277e2bb94 2023-03-19 09:54:42,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/dfa04aacd7fc4c47ab0b7985fda79cea to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/dfa04aacd7fc4c47ab0b7985fda79cea 2023-03-19 09:54:42,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1976e42b44644a53b3580d1cda37152a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/1976e42b44644a53b3580d1cda37152a 2023-03-19 09:54:42,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1f956f5e701e430181210f5e9c3d288a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/991efbaeb0d7413baf9c47a27f8196a0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/757eb84f76ec473ca0cc4d1bc75bceeb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/bf5705e07bb9410d9ba5b213293d6acd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/a447f8f51d81474f89c64b3878c20d49, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/4675bcc8b4f247509ef959e5be87b6cf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9771a61c422c4c689c8bc7776f096cf3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c92e5dfc18c845368ffa46dd3a1ea4d2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/685bc87a7fe04030a7bfe88ce91d4083, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/46ca05b78dce4d95afbad8686fee420a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/8c6a83a8812b49b7a5d82ca307a5a69c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/cc806c2c3a3742c18b222a1db245fd26, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c4236c280c6f4bb2ae8569a5355ce232, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/f741b9b944944c36b6e5500f73ba5134, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c04251f9793e4e8387a640da00878296, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/304ad18cf7fa4746805f0a631d4e7666, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/b32709c3001f4868b63db3e3f82a5594, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/0721f969cb7e49e99f7482a9e397cdf6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/e34c74493c7144e5a3f12ce71f1c2214, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ce1823b1d68745b38e05dbd3ab948cf8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d18f16b9a9614c788608741d93b9876e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/7109159d383044c4bc3c4d63fda8c333] to archive 2023-03-19 09:54:42,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:54:42,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1f956f5e701e430181210f5e9c3d288a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1f956f5e701e430181210f5e9c3d288a 2023-03-19 09:54:42,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/991efbaeb0d7413baf9c47a27f8196a0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/991efbaeb0d7413baf9c47a27f8196a0 2023-03-19 09:54:42,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/757eb84f76ec473ca0cc4d1bc75bceeb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/757eb84f76ec473ca0cc4d1bc75bceeb 2023-03-19 09:54:42,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/bf5705e07bb9410d9ba5b213293d6acd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/bf5705e07bb9410d9ba5b213293d6acd 2023-03-19 09:54:42,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/a447f8f51d81474f89c64b3878c20d49 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/a447f8f51d81474f89c64b3878c20d49 2023-03-19 09:54:42,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/4675bcc8b4f247509ef959e5be87b6cf to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/4675bcc8b4f247509ef959e5be87b6cf 2023-03-19 09:54:42,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9771a61c422c4c689c8bc7776f096cf3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9771a61c422c4c689c8bc7776f096cf3 2023-03-19 09:54:42,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c92e5dfc18c845368ffa46dd3a1ea4d2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c92e5dfc18c845368ffa46dd3a1ea4d2 2023-03-19 09:54:42,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/685bc87a7fe04030a7bfe88ce91d4083 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/685bc87a7fe04030a7bfe88ce91d4083 2023-03-19 09:54:42,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/46ca05b78dce4d95afbad8686fee420a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/46ca05b78dce4d95afbad8686fee420a 2023-03-19 09:54:42,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/8c6a83a8812b49b7a5d82ca307a5a69c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/8c6a83a8812b49b7a5d82ca307a5a69c 2023-03-19 09:54:42,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/cc806c2c3a3742c18b222a1db245fd26 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/cc806c2c3a3742c18b222a1db245fd26 2023-03-19 09:54:42,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c4236c280c6f4bb2ae8569a5355ce232 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c4236c280c6f4bb2ae8569a5355ce232 2023-03-19 09:54:42,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/f741b9b944944c36b6e5500f73ba5134 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/f741b9b944944c36b6e5500f73ba5134 2023-03-19 09:54:42,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c04251f9793e4e8387a640da00878296 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c04251f9793e4e8387a640da00878296 2023-03-19 09:54:42,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/304ad18cf7fa4746805f0a631d4e7666 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/304ad18cf7fa4746805f0a631d4e7666 2023-03-19 09:54:42,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/b32709c3001f4868b63db3e3f82a5594 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/b32709c3001f4868b63db3e3f82a5594 2023-03-19 09:54:42,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/0721f969cb7e49e99f7482a9e397cdf6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/0721f969cb7e49e99f7482a9e397cdf6 2023-03-19 09:54:42,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/e34c74493c7144e5a3f12ce71f1c2214 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/e34c74493c7144e5a3f12ce71f1c2214 2023-03-19 09:54:42,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ce1823b1d68745b38e05dbd3ab948cf8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ce1823b1d68745b38e05dbd3ab948cf8 2023-03-19 09:54:42,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d18f16b9a9614c788608741d93b9876e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d18f16b9a9614c788608741d93b9876e 2023-03-19 09:54:42,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/7109159d383044c4bc3c4d63fda8c333 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/7109159d383044c4bc3c4d63fda8c333 2023-03-19 09:54:42,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/recovered.edits/578.seqid, newMaxSeqId=578, maxSeqId=4 2023-03-19 09:54:42,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4. 2023-03-19 09:54:42,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4a5125924c57b3613b6b6094511223b4: 2023-03-19 09:54:42,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:42,797 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=4a5125924c57b3613b6b6094511223b4, regionState=CLOSED 2023-03-19 09:54:42,797 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219682797"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219682797"}]},"ts":"1679219682797"} 2023-03-19 09:54:42,799 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-19 09:54:42,799 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 4a5125924c57b3613b6b6094511223b4, server=jenkins-hbase4.apache.org,34669,1679219618625 in 3.2590 sec 2023-03-19 09:54:42,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-19 09:54:42,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4a5125924c57b3613b6b6094511223b4, UNASSIGN in 3.2630 sec 2023-03-19 09:54:42,801 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219682801"}]},"ts":"1679219682801"} 2023-03-19 09:54:42,802 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-19 09:54:42,803 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-19 09:54:42,805 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3670 sec 2023-03-19 09:54:43,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-19 09:54:43,545 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-19 09:54:43,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-19 09:54:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:43,548 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-19 09:54:43,549 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:43,553 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:43,555 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/recovered.edits] 2023-03-19 09:54:43,558 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/87c8b3e1c4c746d1938967300058b676 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/87c8b3e1c4c746d1938967300058b676 2023-03-19 09:54:43,559 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/96aea353dca4434d94aba765aaaf2989 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/96aea353dca4434d94aba765aaaf2989 2023-03-19 09:54:43,559 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a47dfe5efada47baa8b5ecf69ecd26b5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a47dfe5efada47baa8b5ecf69ecd26b5 2023-03-19 09:54:43,560 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a513c1eebeb44fe08e7224ce74796419 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/a513c1eebeb44fe08e7224ce74796419 2023-03-19 09:54:43,561 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c3733108b1104f838209fe3f5d22bba7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/A/c3733108b1104f838209fe3f5d22bba7 2023-03-19 09:54:43,563 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/05ba8c78642f487fb473925dd77fc219 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/05ba8c78642f487fb473925dd77fc219 2023-03-19 09:54:43,564 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/11459215d8484047ba85307d895e8434 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/11459215d8484047ba85307d895e8434 2023-03-19 09:54:43,565 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/39e1105102b54cc7a30ca36dd52e7d93 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/39e1105102b54cc7a30ca36dd52e7d93 2023-03-19 09:54:43,566 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/a2d3114c49094b1f8322a5566edef869 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/a2d3114c49094b1f8322a5566edef869 2023-03-19 09:54:43,567 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/eb3ae3ab68fb4f688bfbd3f08e942026 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/B/eb3ae3ab68fb4f688bfbd3f08e942026 2023-03-19 09:54:43,569 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1720613b475440048e7b5d376ceebe52 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/1720613b475440048e7b5d376ceebe52 2023-03-19 09:54:43,570 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9796172bc0924015b73f636b70e05bc0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/9796172bc0924015b73f636b70e05bc0 2023-03-19 09:54:43,571 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ae15ce1642f7454e945fbe8e91567124 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/ae15ce1642f7454e945fbe8e91567124 2023-03-19 09:54:43,572 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c7c8e5acc1574f13bb9d7f000fa7c892 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/c7c8e5acc1574f13bb9d7f000fa7c892 2023-03-19 09:54:43,573 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d0d9dde991f14fdf9961100fb3f78595 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/C/d0d9dde991f14fdf9961100fb3f78595 2023-03-19 09:54:43,576 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/recovered.edits/578.seqid to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4/recovered.edits/578.seqid 2023-03-19 09:54:43,577 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/4a5125924c57b3613b6b6094511223b4 2023-03-19 09:54:43,577 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:54:43,578 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-19 09:54:43,578 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-19 09:54:43,581 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031900138bf324f44784bb180a40449a99b0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031900138bf324f44784bb180a40449a99b0 2023-03-19 09:54:43,582 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195029377d145d4d9db16148d34eae0c2d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195029377d145d4d9db16148d34eae0c2d 2023-03-19 09:54:43,584 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031957b9898be2234290a2ea9eaa778270a6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031957b9898be2234290a2ea9eaa778270a6 2023-03-19 09:54:43,585 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195838168af8264a4baa81e0b461a74755 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195838168af8264a4baa81e0b461a74755 2023-03-19 09:54:43,586 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031962d43d95240d4308b130b104f1d31b43 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031962d43d95240d4308b130b104f1d31b43 2023-03-19 09:54:43,588 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196cb893df15934308820bfc4fcfff1384 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196cb893df15934308820bfc4fcfff1384 2023-03-19 09:54:43,589 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197aaca9dc26674813a8ee2fca72108a6b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197aaca9dc26674813a8ee2fca72108a6b 2023-03-19 09:54:43,590 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197b91f94d19cf471e9c0a04db6493eaad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197b91f94d19cf471e9c0a04db6493eaad 2023-03-19 09:54:43,592 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031981a2629ed88048a1a477ffa4512f0d44 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031981a2629ed88048a1a477ffa4512f0d44 2023-03-19 09:54:43,593 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198567e233938143c7b3c82840cf2219b3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198567e233938143c7b3c82840cf2219b3 2023-03-19 09:54:43,594 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198aaadcefd51a4cc4b21c69d850ebb116 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198aaadcefd51a4cc4b21c69d850ebb116 2023-03-19 09:54:43,596 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319953c8192ccba423489cf9a4df54ae9a3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319953c8192ccba423489cf9a4df54ae9a3 2023-03-19 09:54:43,597 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303199eaa43df96a842499c579e08093f2bbb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303199eaa43df96a842499c579e08093f2bbb 2023-03-19 09:54:43,598 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a6e79e172a9c49b5aba4196dbc4cdf67 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a6e79e172a9c49b5aba4196dbc4cdf67 2023-03-19 09:54:43,599 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b03c7205540a4a21b63711ab01cc5832 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b03c7205540a4a21b63711ab01cc5832 2023-03-19 09:54:43,601 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b3436008a1bd4286ae6ed7a828153c87 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b3436008a1bd4286ae6ed7a828153c87 2023-03-19 09:54:43,602 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b3b6e461bf8847abb4979fd95e755522 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b3b6e461bf8847abb4979fd95e755522 2023-03-19 09:54:43,603 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319c36a8030db4b44c59c0af924b3440ce9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319c36a8030db4b44c59c0af924b3440ce9 2023-03-19 09:54:43,604 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e1de6627bb0a4459bcf8ba0ff2c2e6ad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e1de6627bb0a4459bcf8ba0ff2c2e6ad 2023-03-19 09:54:43,605 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e4420c264b9b4e0595b26408739f6932 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e4420c264b9b4e0595b26408739f6932 2023-03-19 09:54:43,607 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e91dd66c57ea4831a290c1f4f9f29229 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e91dd66c57ea4831a290c1f4f9f29229 2023-03-19 09:54:43,608 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319fbd00e4b859c41d8acc3a7b47a5bbb5d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319fbd00e4b859c41d8acc3a7b47a5bbb5d 2023-03-19 09:54:43,609 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-19 09:54:43,611 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:43,613 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-19 09:54:43,615 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-19 09:54:43,616 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:43,616 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-19 09:54:43,616 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679219683616"}]},"ts":"9223372036854775807"} 2023-03-19 09:54:43,618 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-19 09:54:43,618 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4a5125924c57b3613b6b6094511223b4, NAME => 'TestAcidGuarantees,,1679219654743.4a5125924c57b3613b6b6094511223b4.', STARTKEY => '', ENDKEY => ''}] 2023-03-19 09:54:43,618 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-19 09:54:43,618 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679219683618"}]},"ts":"9223372036854775807"} 2023-03-19 09:54:43,619 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-19 09:54:43,621 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:43,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 75 msec 2023-03-19 09:54:43,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-19 09:54:43,649 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-19 09:54:43,663 INFO [Listener at localhost/45961] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 231), OpenFileDescriptor=559 (was 530) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=429 (was 445), ProcessCount=169 (was 169), AvailableMemoryMB=1626 (was 1839) 2023-03-19 09:54:43,675 INFO [Listener at localhost/45961] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=429, ProcessCount=169, AvailableMemoryMB=1625 2023-03-19 09:54:43,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:54:43,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:54:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-19 09:54:43,680 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-19 09:54:43,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-19 09:54:43,681 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-19 09:54:43,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-19 09:54:43,682 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:43,683 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0 empty. 2023-03-19 09:54:43,683 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:43,683 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:54:43,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-19 09:54:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-19 09:54:44,091 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:54:44,093 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8dad03d10fa8d87944de8392ea9480f0, NAME => 'TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp 2023-03-19 09:54:44,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-19 09:54:44,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:54:44,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8dad03d10fa8d87944de8392ea9480f0, disabling compactions & flushes 2023-03-19 09:54:44,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. after waiting 0 ms 2023-03-19 09:54:44,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:44,501 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-19 09:54:44,502 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219684502"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219684502"}]},"ts":"1679219684502"} 2023-03-19 09:54:44,503 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-19 09:54:44,504 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-19 09:54:44,504 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219684504"}]},"ts":"1679219684504"} 2023-03-19 09:54:44,504 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-19 09:54:44,509 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dad03d10fa8d87944de8392ea9480f0, ASSIGN}] 2023-03-19 09:54:44,510 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dad03d10fa8d87944de8392ea9480f0, ASSIGN 2023-03-19 09:54:44,511 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dad03d10fa8d87944de8392ea9480f0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=false 2023-03-19 09:54:44,662 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=8dad03d10fa8d87944de8392ea9480f0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:44,662 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219684662"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219684662"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219684662"}]},"ts":"1679219684662"} 2023-03-19 09:54:44,663 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:54:44,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-19 09:54:44,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8dad03d10fa8d87944de8392ea9480f0, NAME => 'TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:54:44,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:54:44,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,820 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,822 DEBUG [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A 2023-03-19 09:54:44,822 DEBUG [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A 2023-03-19 09:54:44,822 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:44,822 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dad03d10fa8d87944de8392ea9480f0 columnFamilyName A 2023-03-19 09:54:44,823 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.HStore(310): Store=8dad03d10fa8d87944de8392ea9480f0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:44,823 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,824 DEBUG [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B 2023-03-19 09:54:44,824 DEBUG [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B 2023-03-19 09:54:44,824 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:44,825 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dad03d10fa8d87944de8392ea9480f0 columnFamilyName B 2023-03-19 09:54:44,825 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.HStore(310): Store=8dad03d10fa8d87944de8392ea9480f0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:44,825 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,826 DEBUG [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C 2023-03-19 09:54:44,826 DEBUG [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C 2023-03-19 09:54:44,827 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:54:44,827 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dad03d10fa8d87944de8392ea9480f0 columnFamilyName C 2023-03-19 09:54:44,827 INFO [StoreOpener-8dad03d10fa8d87944de8392ea9480f0-1] regionserver.HStore(310): Store=8dad03d10fa8d87944de8392ea9480f0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:54:44,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:54:44,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:44,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:54:44,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8dad03d10fa8d87944de8392ea9480f0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68616242, jitterRate=0.022461682558059692}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:54:44,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:44,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., pid=30, masterSystemTime=1679219684815 2023-03-19 09:54:44,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:44,835 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=8dad03d10fa8d87944de8392ea9480f0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:44,835 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219684835"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219684835"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219684835"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219684835"}]},"ts":"1679219684835"} 2023-03-19 09:54:44,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-19 09:54:44,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 in 173 msec 2023-03-19 09:54:44,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-19 09:54:44,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dad03d10fa8d87944de8392ea9480f0, ASSIGN in 329 msec 2023-03-19 09:54:44,839 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-19 09:54:44,840 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219684840"}]},"ts":"1679219684840"} 2023-03-19 09:54:44,840 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-19 09:54:44,842 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-19 09:54:44,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-03-19 09:54:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-19 09:54:45,786 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-19 09:54:45,786 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,791 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,792 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,794 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35894, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,795 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-19 09:54:45,796 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35312, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-19 09:54:45,798 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,801 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,801 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,804 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,805 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,809 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,809 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,812 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,812 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,815 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,816 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,819 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,820 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,824 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,824 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,828 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,828 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,831 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,832 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:54:45,835 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:54:45,841 DEBUG [hconnection-0x6e1cf94c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,844 DEBUG [hconnection-0x33797c86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,844 DEBUG [hconnection-0x7b57f255-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,844 DEBUG [hconnection-0x49a38823-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,844 DEBUG [hconnection-0x704198d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,845 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35918, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,849 DEBUG [hconnection-0x762771d9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,849 DEBUG [hconnection-0x2f40f6b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,849 DEBUG [hconnection-0x2885e839-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,849 DEBUG [hconnection-0x41c44002-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,850 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,850 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35960, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:45,854 DEBUG [hconnection-0x7410e40d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:54:45,857 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35912, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,857 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,858 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35936, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,858 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35930, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,858 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35948, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,858 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35946, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,861 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35970, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:54:45,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:45,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:45,862 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:45,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:45,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:45,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:45,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:45,870 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:45,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:45,870 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:45,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:45,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:45,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:45,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:45,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:45,873 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:45,873 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-19 09:54:45,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:45,873 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:45,873 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:45,874 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:45,874 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:45,874 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:45,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:45,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:45,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:45,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:45,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:45,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@694ff336[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:45,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:45,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:45,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:45,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:45,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:45,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:45,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:45,882 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:45,886 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:45,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:45,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:45,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:45,886 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:45,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:45,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:45,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:45,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,892 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:45,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:45,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:45,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:45,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:45,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:45,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:45,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,896 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,896 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:45,896 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:45,896 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:45,896 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:45,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:45,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219745898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679219685886, max timestamp=1679219685901Num uniques 859; 2023-03-19 09:54:45,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:54:45,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219685894, max timestamp=1679219685901Num uniques 859; 2023-03-19 09:54:45,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-19 09:54:45,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219685894, max timestamp=1679219685901Num uniques 859; 2023-03-19 09:54:45,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-19 09:54:45,913 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/3c17e5ea49d54856b35e18ac94e82007 2023-03-19 09:54:45,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219745919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219745920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219745920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:45,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219745922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:45,996 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:45,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:45,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219746004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,004 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,009 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,009 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,009 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:46,009 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,011 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d83c7d8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,018 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,019 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,019 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,020 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,024 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,024 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219746022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219746023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219746024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,026 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,026 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,026 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,026 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219746025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:46,031 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,031 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,031 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,031 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:46,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:46,031 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:46,031 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:46,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,131 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:46,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:46,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,135 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,136 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,136 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,138 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,138 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,138 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,138 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,139 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:46,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,142 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cffd74b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,145 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,145 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,145 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,146 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,146 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,149 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,149 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,157 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:46,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,161 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:46,162 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:46,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:46,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219746206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219746227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219746227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219746227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219746228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,262 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:46,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:46,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,265 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,270 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:46,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,272 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a01755[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,278 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,287 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:46,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:46,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,287 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,287 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:46,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,347 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/1e10f643c3554d239247a4d131579751 2023-03-19 09:54:46,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/41528741ba0d498b8cf1974ab3d2dbaf 2023-03-19 09:54:46,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/3c17e5ea49d54856b35e18ac94e82007 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/3c17e5ea49d54856b35e18ac94e82007 2023-03-19 09:54:46,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/3c17e5ea49d54856b35e18ac94e82007, entries=150, sequenceid=14, filesize=11.6 K 2023-03-19 09:54:46,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/1e10f643c3554d239247a4d131579751 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e10f643c3554d239247a4d131579751 2023-03-19 09:54:46,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e10f643c3554d239247a4d131579751, entries=150, sequenceid=14, filesize=11.6 K 2023-03-19 09:54:46,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/41528741ba0d498b8cf1974ab3d2dbaf as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/41528741ba0d498b8cf1974ab3d2dbaf 2023-03-19 09:54:46,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/41528741ba0d498b8cf1974ab3d2dbaf, entries=150, sequenceid=14, filesize=11.6 K 2023-03-19 09:54:46,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=111.03 KB/113692 for 8dad03d10fa8d87944de8392ea9480f0 in 505ms, sequenceid=14, compaction requested=false 2023-03-19 09:54:46,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:46,388 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:46,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:46,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,390 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,395 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,395 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,395 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=111.03 KB heapSize=313.63 KB 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:46,395 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:46,410 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/024e7adbeafe4ae5a6f0e8c53aa4dd40 2023-03-19 09:54:46,434 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.39 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/b3f84ed1aafb42889330a87c0eb723cb 2023-03-19 09:54:46,449 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.39 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/8a9048b06612443db0f59f1a22aa7cfc 2023-03-19 09:54:46,453 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/024e7adbeafe4ae5a6f0e8c53aa4dd40 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/024e7adbeafe4ae5a6f0e8c53aa4dd40 2023-03-19 09:54:46,456 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/024e7adbeafe4ae5a6f0e8c53aa4dd40, entries=150, sequenceid=47, filesize=11.6 K 2023-03-19 09:54:46,457 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/b3f84ed1aafb42889330a87c0eb723cb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b3f84ed1aafb42889330a87c0eb723cb 2023-03-19 09:54:46,461 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b3f84ed1aafb42889330a87c0eb723cb, entries=150, sequenceid=47, filesize=11.6 K 2023-03-19 09:54:46,461 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/8a9048b06612443db0f59f1a22aa7cfc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/8a9048b06612443db0f59f1a22aa7cfc 2023-03-19 09:54:46,471 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/8a9048b06612443db0f59f1a22aa7cfc, entries=150, sequenceid=47, filesize=11.6 K 2023-03-19 09:54:46,472 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~111.03 KB/113692, heapSize ~313.58 KB/321104, currentSize=0 B/0 for 8dad03d10fa8d87944de8392ea9480f0 in 77ms, sequenceid=47, compaction requested=false 2023-03-19 09:54:46,472 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:46,472 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54608a74[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,477 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,477 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,478 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,490 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,490 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,492 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,492 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,492 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,494 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:46,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:46,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:46,497 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:46,497 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:46,522 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:54:46,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:46,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:46,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:46,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:46,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:46,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:46,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219686537, max timestamp=1679219686541Num uniques 859; 2023-03-19 09:54:46,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:46,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219686537, max timestamp=1679219686541Num uniques 859; 2023-03-19 09:54:46,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:46,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219686537, max timestamp=1679219686541Num uniques 859; 2023-03-19 09:54:46,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219746552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219746552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219746553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219746553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219746556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,597 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:46,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:46,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,601 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,603 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:46,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,608 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57de3a27[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,610 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,610 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,611 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,611 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,611 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,611 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,615 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,615 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,617 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,617 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,617 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,620 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:46,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,624 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,625 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219746657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219746657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219746657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219746657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219746660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,725 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:46,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:46,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,728 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,730 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:46,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7539fa18[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,739 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,742 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,742 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,742 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,742 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,742 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,745 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:46,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,748 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,748 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,848 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:46,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:46,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,851 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,854 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,854 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:46,855 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,858 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,859 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,859 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37614155[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219746860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219746861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219746861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219746862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:46,866 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219746864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,866 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,866 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,868 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,869 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,870 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,870 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,870 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:46,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:46,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,880 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:46,880 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:46,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:46,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/9d80f2c6f5a541958fe890f837afac69 2023-03-19 09:54:46,980 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:46,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:46,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:46,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:46,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:46,983 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:46,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:46,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:46,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:46,985 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:46,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:46,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:46,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:46,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:46,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:46,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:46,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:46,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:46,987 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:46,987 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,987 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:46,987 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:46,987 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:46,989 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67024ee7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:46,991 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:46,991 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,992 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:46,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:46,995 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:46,997 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:46,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:46,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:46,997 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:46,997 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:46,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:46,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:46,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:46,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:46,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:47,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:47,003 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:47,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,003 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:47,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,004 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,104 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:47,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:47,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:47,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:47,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:47,107 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:47,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:47,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:47,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,108 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:47,110 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,110 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:47,110 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:47,110 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:47,115 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ab5f22[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:47,115 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:47,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:47,116 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:47,116 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,116 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:47,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:47,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:47,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:47,120 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:47,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:47,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:47,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:47,120 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:47,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:47,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,130 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,130 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:47,131 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:47,130 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219747165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219747165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219747166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219747168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219747169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,231 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:47,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:47,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:47,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:47,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:47,234 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:47,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:47,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:47,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:47,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:47,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:47,238 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,238 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:47,238 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:47,238 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:47,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b788435[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:47,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:47,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:47,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:47,246 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:47,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:47,249 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:47,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:47,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:47,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:47,249 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:47,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,251 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:47,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,257 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:47,257 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:47,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:47,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/ad1aa295760d45c38aa4922a07f46e6e 2023-03-19 09:54:47,357 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:47,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:47,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:47,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:47,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:47,360 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:47,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:47,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:47,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:47,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:47,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:47,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:47,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:47,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:47,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:47,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ae09e6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:47,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:47,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,370 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:47,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:47,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:47,374 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:47,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:47,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:47,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:47,374 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:47,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,383 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:47,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:47,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:47,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,384 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,384 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:47,384 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,384 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,384 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:47,384 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,484 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:47,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:47,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:47,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:47,488 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:47,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:47,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:47,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,489 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:47,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:47,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:47,491 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,491 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:47,491 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:47,491 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:47,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:47,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332e12a2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:47,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:47,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,498 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:47,499 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:47,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,499 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,499 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:47,501 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:47,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:47,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:47,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:47,502 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:47,503 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,503 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,503 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:47,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:47,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,508 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:47,508 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:47,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,608 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:47,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:47,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:47,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:47,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:47,611 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:47,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:47,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:47,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:47,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:47,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:47,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:47,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:47,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:47,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:47,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286709aa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:47,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:47,623 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,625 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:47,625 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:47,625 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:47,625 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,625 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:47,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:47,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:47,634 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:47,634 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:47,634 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:47,634 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:47,634 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:47,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,640 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,644 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,644 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,644 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:47,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:47,644 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,644 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,644 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,645 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,645 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,645 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,645 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:47,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,645 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:47,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219747668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219747669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219747672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219747673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219747674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,745 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:47,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:47,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:47,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:47,748 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:47,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:47,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:47,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/d3761806ba5942e383e5954f5153e424 2023-03-19 09:54:47,753 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:47,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:47,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:47,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:47,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:47,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:47,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/9d80f2c6f5a541958fe890f837afac69 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d80f2c6f5a541958fe890f837afac69 2023-03-19 09:54:47,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:47,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:47,772 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,772 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:47,773 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:47,773 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,773 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:47,773 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:47,773 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,773 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:47,773 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:47,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d80f2c6f5a541958fe890f837afac69, entries=200, sequenceid=57, filesize=13.9 K 2023-03-19 09:54:47,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/ad1aa295760d45c38aa4922a07f46e6e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ad1aa295760d45c38aa4922a07f46e6e 2023-03-19 09:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@178179bf[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:47,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:47,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ad1aa295760d45c38aa4922a07f46e6e, entries=150, sequenceid=57, filesize=11.6 K 2023-03-19 09:54:47,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:47,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/d3761806ba5942e383e5954f5153e424 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d3761806ba5942e383e5954f5153e424 2023-03-19 09:54:47,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:47,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d3761806ba5942e383e5954f5153e424, entries=150, sequenceid=57, filesize=11.6 K 2023-03-19 09:54:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8dad03d10fa8d87944de8392ea9480f0 in 1284ms, sequenceid=57, compaction requested=true 2023-03-19 09:54:47,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:47,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:47,807 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:47,807 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:47,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:47,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,808 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:47,808 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:54:47,808 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:47,808 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:47,808 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:47,808 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,808 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/3c17e5ea49d54856b35e18ac94e82007, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/024e7adbeafe4ae5a6f0e8c53aa4dd40, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d80f2c6f5a541958fe890f837afac69] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=37.1 K 2023-03-19 09:54:47,808 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:47,808 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:47,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:47,808 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:47,808 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:54:47,809 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,809 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3c17e5ea49d54856b35e18ac94e82007, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679219685866 2023-03-19 09:54:47,809 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e10f643c3554d239247a4d131579751, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b3f84ed1aafb42889330a87c0eb723cb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ad1aa295760d45c38aa4922a07f46e6e] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=34.8 K 2023-03-19 09:54:47,809 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 024e7adbeafe4ae5a6f0e8c53aa4dd40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679219685911 2023-03-19 09:54:47,809 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1e10f643c3554d239247a4d131579751, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679219685866 2023-03-19 09:54:47,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9d80f2c6f5a541958fe890f837afac69, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679219686513 2023-03-19 09:54:47,810 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b3f84ed1aafb42889330a87c0eb723cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679219685911 2023-03-19 09:54:47,810 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ad1aa295760d45c38aa4922a07f46e6e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679219686518 2023-03-19 09:54:47,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,814 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:47,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:47,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:47,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:47,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:47,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:47,829 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:47,829 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:47,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:47,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:47,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:47,843 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#205 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:47,845 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#206 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:47,929 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:47,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:47,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:47,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:47,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:47,933 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:47,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:47,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:47,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:47,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:47,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:47,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:47,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:47,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:47,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:47,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:47,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:47,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:47,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:47,948 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:47,948 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:47,948 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-19 09:54:47,948 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:47,949 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:47,949 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:47,949 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:47,949 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:47,949 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:48,260 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/8848450d1d8342ddbd6aa5d90273a94b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8848450d1d8342ddbd6aa5d90273a94b 2023-03-19 09:54:48,262 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/f25ab462e30d420bbc68dbd7d58488fd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/f25ab462e30d420bbc68dbd7d58488fd 2023-03-19 09:54:48,265 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 8848450d1d8342ddbd6aa5d90273a94b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:48,265 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:48,265 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=13, startTime=1679219687806; duration=0sec 2023-03-19 09:54:48,265 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:48,265 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:48,267 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:48,267 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:54:48,267 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:48,267 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into f25ab462e30d420bbc68dbd7d58488fd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:48,267 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/41528741ba0d498b8cf1974ab3d2dbaf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/8a9048b06612443db0f59f1a22aa7cfc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d3761806ba5942e383e5954f5153e424] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=34.8 K 2023-03-19 09:54:48,267 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:48,267 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=13, startTime=1679219687807; duration=0sec 2023-03-19 09:54:48,267 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:48,268 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 41528741ba0d498b8cf1974ab3d2dbaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679219685866 2023-03-19 09:54:48,268 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8a9048b06612443db0f59f1a22aa7cfc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679219685911 2023-03-19 09:54:48,268 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d3761806ba5942e383e5954f5153e424, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679219686518 2023-03-19 09:54:48,273 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:48,281 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/e5ec305e09f04f439d805d9d56b525a4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/e5ec305e09f04f439d805d9d56b525a4 2023-03-19 09:54:48,286 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into e5ec305e09f04f439d805d9d56b525a4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:48,286 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:48,286 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=13, startTime=1679219687807; duration=0sec 2023-03-19 09:54:48,286 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:48,349 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-19 09:54:48,365 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/633ce4e581c74ad3b3c33f0a0f451057 2023-03-19 09:54:48,382 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/ace63dd026204c6ea6aff39bca1af86c 2023-03-19 09:54:48,396 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/6e339cb2942b4939b625c1318d48471f 2023-03-19 09:54:48,400 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/633ce4e581c74ad3b3c33f0a0f451057 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/633ce4e581c74ad3b3c33f0a0f451057 2023-03-19 09:54:48,405 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/633ce4e581c74ad3b3c33f0a0f451057, entries=150, sequenceid=93, filesize=11.6 K 2023-03-19 09:54:48,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/ace63dd026204c6ea6aff39bca1af86c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ace63dd026204c6ea6aff39bca1af86c 2023-03-19 09:54:48,416 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ace63dd026204c6ea6aff39bca1af86c, entries=150, sequenceid=93, filesize=11.6 K 2023-03-19 09:54:48,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/6e339cb2942b4939b625c1318d48471f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6e339cb2942b4939b625c1318d48471f 2023-03-19 09:54:48,422 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6e339cb2942b4939b625c1318d48471f, entries=150, sequenceid=93, filesize=11.6 K 2023-03-19 09:54:48,422 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 8dad03d10fa8d87944de8392ea9480f0 in 474ms, sequenceid=93, compaction requested=false 2023-03-19 09:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:48,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:48,426 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:48,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d6b390[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:48,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:48,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,431 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:48,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:48,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:48,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:48,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:48,436 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:48,436 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:48,436 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:48,436 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:48,436 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:48,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:48,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:48,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,444 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:48,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:48,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:48,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:48,444 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:48,444 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:48,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:48,544 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:48,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:48,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:48,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:48,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:48,548 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:48,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:48,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:48,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:48,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,551 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:48,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:48,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:48,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:48,553 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:48,553 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:48,553 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:48,553 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:48,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@622e0e0a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:48,557 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:48,557 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,558 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:48,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:48,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:48,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:48,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:48,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:48,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:48,563 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:48,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:48,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:48,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:48,563 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:48,566 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,566 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:48,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,566 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:48,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:48,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:48,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:48,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:48,570 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:48,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:48,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:48,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:48,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:48,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:48,571 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,571 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,571 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:48,571 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:48,571 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:48,571 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:48,671 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:48,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:48,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:48,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:48,675 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:48,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:48,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:48,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:48,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,677 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:48,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:48,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:48,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:48,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:48,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:48,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:48,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:48,679 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:48,679 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:48,679 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-19 09:54:48,679 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:48,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:48,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:48,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:48,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:48,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:48,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:48,682 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:48,688 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/c823958f705f41be8975b28c2413bf46 2023-03-19 09:54:48,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679219688681, max timestamp=1679219688693Num uniques 859; 2023-03-19 09:54:48,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-19 09:54:48,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679219688681, max timestamp=1679219688693Num uniques 859; 2023-03-19 09:54:48,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679219688681, max timestamp=1679219688693Num uniques 859; 2023-03-19 09:54:48,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-19 09:54:48,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-19 09:54:48,699 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/a2ecc67d45aa4078a106bbdad659e8e0 2023-03-19 09:54:48,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679219688707, max timestamp=1679219688710Num uniques 1168; 2023-03-19 09:54:48,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-03-19 09:54:48,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679219688707, max timestamp=1679219688710Num uniques 1168; 2023-03-19 09:54:48,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-03-19 09:54:48,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679219688707, max timestamp=1679219688710Num uniques 1168; 2023-03-19 09:54:48,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-03-19 09:54:48,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:48,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219748723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219688720, max timestamp=1679219688724Num uniques 1077; 2023-03-19 09:54:48,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-19 09:54:48,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219688720, max timestamp=1679219688724Num uniques 1077; 2023-03-19 09:54:48,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-19 09:54:48,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219688720, max timestamp=1679219688724Num uniques 1077; 2023-03-19 09:54:48,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-19 09:54:48,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:48,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219748738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:48,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219748739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679219688735, max timestamp=1679219688739Num uniques 1136; 2023-03-19 09:54:48,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-19 09:54:48,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679219688735, max timestamp=1679219688739Num uniques 1136; 2023-03-19 09:54:48,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679219688735, max timestamp=1679219688739Num uniques 1136; 2023-03-19 09:54:48,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-19 09:54:48,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-19 09:54:48,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679219688751, max timestamp=1679219688786Num uniques 1095; 2023-03-19 09:54:48,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-19 09:54:48,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679219688751, max timestamp=1679219688786Num uniques 1095; 2023-03-19 09:54:48,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-19 09:54:48,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679219688751, max timestamp=1679219688786Num uniques 1095; 2023-03-19 09:54:48,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-19 09:54:48,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219688797, max timestamp=1679219688844Num uniques 1104; 2023-03-19 09:54:48,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219688797, max timestamp=1679219688844Num uniques 1104; 2023-03-19 09:54:48,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-19 09:54:48,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-19 09:54:48,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219688797, max timestamp=1679219688844Num uniques 1054; 2023-03-19 09:54:48,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-19 09:54:48,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1679219688863, max timestamp=1679219688869Num uniques 1013; 2023-03-19 09:54:48,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679219688863, max timestamp=1679219688869Num uniques 1013; 2023-03-19 09:54:48,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-19 09:54:48,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1679219688863, max timestamp=1679219688869Num uniques 1013; 2023-03-19 09:54:48,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=9745, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-19 09:54:48,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=9745, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-19 09:54:48,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679219688882, max timestamp=1679219688888Num uniques 1072; 2023-03-19 09:54:48,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-19 09:54:48,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679219688882, max timestamp=1679219688888Num uniques 1072; 2023-03-19 09:54:48,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-19 09:54:48,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679219688882, max timestamp=1679219688888Num uniques 1072; 2023-03-19 09:54:48,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-19 09:54:48,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679219688900, max timestamp=1679219688909Num uniques 1031; 2023-03-19 09:54:48,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-19 09:54:48,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679219688900, max timestamp=1679219688909Num uniques 1031; 2023-03-19 09:54:48,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-19 09:54:48,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679219688900, max timestamp=1679219688909Num uniques 1031; 2023-03-19 09:54:48,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-19 09:54:48,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679219688918, max timestamp=1679219688923Num uniques 1040; 2023-03-19 09:54:48,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-03-19 09:54:48,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679219688918, max timestamp=1679219688923Num uniques 1040; 2023-03-19 09:54:48,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-03-19 09:54:48,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679219688918, max timestamp=1679219688923Num uniques 1040; 2023-03-19 09:54:48,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-03-19 09:54:48,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:48,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219748938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1679219688935, max timestamp=1679219688941Num uniques 1149; 2023-03-19 09:54:48,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=15984, suffix heap size=147464,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=349 2023-03-19 09:54:48,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1679219688935, max timestamp=1679219688941Num uniques 1149; 2023-03-19 09:54:48,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=15984, suffix heap size=147464,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=349 2023-03-19 09:54:48,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1679219688935, max timestamp=1679219688941Num uniques 1149; 2023-03-19 09:54:48,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=15984, suffix heap size=147464,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=349 2023-03-19 09:54:48,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:48,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219748959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:48,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679219688956, max timestamp=1679219688961Num uniques 1208; 2023-03-19 09:54:48,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55318, new segment data size=11808, suffix heap size=153600,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=258 2023-03-19 09:54:48,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679219688956, max timestamp=1679219688961Num uniques 1208; 2023-03-19 09:54:48,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=55318, new segment data size=11808, suffix heap size=153600,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=258 2023-03-19 09:54:48,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679219688956, max timestamp=1679219688961Num uniques 1208; 2023-03-19 09:54:48,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55318, new segment data size=11808, suffix heap size=153600,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=258 2023-03-19 09:54:49,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679219688998, max timestamp=1679219689004Num uniques 1117; 2023-03-19 09:54:49,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-03-19 09:54:49,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679219688998, max timestamp=1679219689004Num uniques 1117; 2023-03-19 09:54:49,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-03-19 09:54:49,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679219688998, max timestamp=1679219689004Num uniques 1117; 2023-03-19 09:54:49,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-03-19 09:54:49,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679219689023, max timestamp=1679219689047Num uniques 1076; 2023-03-19 09:54:49,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-19 09:54:49,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679219689023, max timestamp=1679219689047Num uniques 1076; 2023-03-19 09:54:49,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-19 09:54:49,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219689023, max timestamp=1679219689047Num uniques 1076; 2023-03-19 09:54:49,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=6870, suffix heap size=139872,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=150 2023-03-19 09:54:49,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1679219689067, max timestamp=1679219689072Num uniques 1035; 2023-03-19 09:54:49,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-03-19 09:54:49,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1679219689067, max timestamp=1679219689072Num uniques 1035; 2023-03-19 09:54:49,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-03-19 09:54:49,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679219689070, max timestamp=1679219689072Num uniques 1009; 2023-03-19 09:54:49,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8470, suffix heap size=132904,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=185 2023-03-19 09:54:49,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679219689077, max timestamp=1679219689088Num uniques 1144; 2023-03-19 09:54:49,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-03-19 09:54:49,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1679219689077, max timestamp=1679219689088Num uniques 1144; 2023-03-19 09:54:49,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-03-19 09:54:49,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679219689077, max timestamp=1679219689089Num uniques 1044; 2023-03-19 09:54:49,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-19 09:54:49,108 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/d808f211a6a2405d9be1311f78f14ada 2023-03-19 09:54:49,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219749107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679219689099, max timestamp=1679219689111Num uniques 1203; 2023-03-19 09:54:49,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=7005, suffix heap size=153080,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=153 2023-03-19 09:54:49,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679219689099, max timestamp=1679219689111Num uniques 1053; 2023-03-19 09:54:49,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679219689099, max timestamp=1679219689111Num uniques 1053; 2023-03-19 09:54:49,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-19 09:54:49,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-19 09:54:49,114 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/c823958f705f41be8975b28c2413bf46 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c823958f705f41be8975b28c2413bf46 2023-03-19 09:54:49,120 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c823958f705f41be8975b28c2413bf46, entries=100, sequenceid=104, filesize=9.3 K 2023-03-19 09:54:49,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/a2ecc67d45aa4078a106bbdad659e8e0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a2ecc67d45aa4078a106bbdad659e8e0 2023-03-19 09:54:49,132 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a2ecc67d45aa4078a106bbdad659e8e0, entries=100, sequenceid=104, filesize=9.3 K 2023-03-19 09:54:49,133 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/d808f211a6a2405d9be1311f78f14ada as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d808f211a6a2405d9be1311f78f14ada 2023-03-19 09:54:49,138 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d808f211a6a2405d9be1311f78f14ada, entries=100, sequenceid=104, filesize=9.3 K 2023-03-19 09:54:49,139 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=107.34 KB/109920 for 8dad03d10fa8d87944de8392ea9480f0 in 460ms, sequenceid=104, compaction requested=true 2023-03-19 09:54:49,139 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:49,139 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,139 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:49,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:49,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:49,140 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:49,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721471bc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:49,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:49,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,146 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:49,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:49,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:49,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:49,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:49,148 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,148 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=120.76 KB heapSize=345.70 KB 2023-03-19 09:54:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:49,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:49,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:49,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:49,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:49,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:49,153 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:49,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:49,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:49,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:49,153 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:49,155 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,155 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,155 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,156 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,160 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,160 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,160 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,160 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:49,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:49,160 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,160 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,161 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:49,161 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:49,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:49,161 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219749158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219749158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219749159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219749162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219749213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,261 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:49,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:49,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:49,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:49,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:49,264 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:49,264 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,264 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:49,264 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:49,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219749264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219749264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219749264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,267 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219749266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:49,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:49,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:49,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:49,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:49,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:49,274 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:49,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec4b2c0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:49,276 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:49,276 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:49,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:49,278 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:49,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:49,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:49,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:49,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:49,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:49,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:49,284 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,284 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,293 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,294 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:49,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,294 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,294 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:49,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,395 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:49,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:49,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:49,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:49,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:49,398 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:49,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:49,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:49,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,399 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,399 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:49,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:49,401 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,401 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:49,401 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:49,402 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:49,403 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,403 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:49,405 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1ad6bb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:49,405 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,408 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:49,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:49,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:49,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:49,412 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:49,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:49,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:49,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:49,413 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:49,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,415 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219749417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:49,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:49,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,419 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:49,420 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:49,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219749468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219749468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219749469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35946 deadline: 1679219749470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,520 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:49,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:49,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:49,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:49,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:49,523 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:49,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:49,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:49,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,525 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:49,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:49,527 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,527 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:49,527 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:49,527 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:49,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:49,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:49,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6856d9ac[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:49,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,532 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:49,533 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:49,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:49,541 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:49,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:49,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:49,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:49,541 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:49,543 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,543 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,543 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,545 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:49,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:49,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:49,548 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:49,548 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:49,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,566 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/b234541df3f04e2ba7fc2486f8a578b9 2023-03-19 09:54:49,648 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:49,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:49,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:49,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:49,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:49,651 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:49,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:49,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:49,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,653 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,653 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,653 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:49,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:49,654 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,655 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:49,655 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:49,655 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:49,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:49,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391edc8b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:49,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:49,662 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,662 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:49,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:49,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:49,671 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:49,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:49,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:49,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:49,671 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:49,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,675 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,675 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,676 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:49,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:49,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,680 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:49,680 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,681 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219749721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219749770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219749773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219749775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:49,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219749776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,780 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:49,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:49,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:49,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:49,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:49,786 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:49,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:49,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:49,787 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,787 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,788 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:49,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:49,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:49,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:49,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:49,792 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,792 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:49,792 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:49,793 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:49,795 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,795 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,795 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,795 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:49,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72872bda[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:49,800 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:49,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,806 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:49,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:49,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:49,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:49,809 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:49,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:49,812 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:49,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:49,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:49,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:49,813 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:49,815 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,815 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,820 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:49,824 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:49,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:49,824 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:49,825 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:49,932 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:49,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:49,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:49,936 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:49,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:49,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:49,937 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,937 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,938 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:49,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:49,940 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,940 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:49,940 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:49,940 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:49,943 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:49,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e84843f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:49,945 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:49,945 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,946 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:49,947 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,947 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:49,948 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:49,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:49,951 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:49,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:49,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:49,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:49,951 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:49,953 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,953 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,953 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:49,953 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:49,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:49,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:49,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:49,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:49,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:49,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:49,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:49,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:49,962 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:49,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:49,962 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:49,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:49,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:49,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/88c30d8d5538405596c5d8b95bb8e4c8 2023-03-19 09:54:49,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/f3200995ac784716aa909850a497d7c7 2023-03-19 09:54:49,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/b234541df3f04e2ba7fc2486f8a578b9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b234541df3f04e2ba7fc2486f8a578b9 2023-03-19 09:54:50,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b234541df3f04e2ba7fc2486f8a578b9, entries=150, sequenceid=415, filesize=11.9 K 2023-03-19 09:54:50,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/88c30d8d5538405596c5d8b95bb8e4c8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/88c30d8d5538405596c5d8b95bb8e4c8 2023-03-19 09:54:50,006 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/88c30d8d5538405596c5d8b95bb8e4c8, entries=150, sequenceid=415, filesize=11.9 K 2023-03-19 09:54:50,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/f3200995ac784716aa909850a497d7c7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f3200995ac784716aa909850a497d7c7 2023-03-19 09:54:50,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f3200995ac784716aa909850a497d7c7, entries=150, sequenceid=415, filesize=11.9 K 2023-03-19 09:54:50,017 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.58 KB/374352, currentSize=53.67 KB/54960 for 8dad03d10fa8d87944de8392ea9480f0 in 869ms, sequenceid=415, compaction requested=true 2023-03-19 09:54:50,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:50,017 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:50,017 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:50,017 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:50,017 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:50,017 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:50,018 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45603 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:50,018 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45603 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:50,018 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:54:50,018 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:54:50,018 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,018 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,018 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/f25ab462e30d420bbc68dbd7d58488fd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ace63dd026204c6ea6aff39bca1af86c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a2ecc67d45aa4078a106bbdad659e8e0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/88c30d8d5538405596c5d8b95bb8e4c8] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=44.5 K 2023-03-19 09:54:50,018 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8848450d1d8342ddbd6aa5d90273a94b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/633ce4e581c74ad3b3c33f0a0f451057, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c823958f705f41be8975b28c2413bf46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b234541df3f04e2ba7fc2486f8a578b9] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=44.5 K 2023-03-19 09:54:50,019 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f25ab462e30d420bbc68dbd7d58488fd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679219686518 2023-03-19 09:54:50,019 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ace63dd026204c6ea6aff39bca1af86c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679219686550 2023-03-19 09:54:50,019 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a2ecc67d45aa4078a106bbdad659e8e0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1679219688678 2023-03-19 09:54:50,020 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 88c30d8d5538405596c5d8b95bb8e4c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219689124 2023-03-19 09:54:50,020 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8848450d1d8342ddbd6aa5d90273a94b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679219686518 2023-03-19 09:54:50,020 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 633ce4e581c74ad3b3c33f0a0f451057, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679219686550 2023-03-19 09:54:50,021 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c823958f705f41be8975b28c2413bf46, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1679219688678 2023-03-19 09:54:50,022 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b234541df3f04e2ba7fc2486f8a578b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219689124 2023-03-19 09:54:50,030 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:50,031 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:50,062 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:50,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:50,064 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/8a790cbed1774b39be41d1434d4604f9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8a790cbed1774b39be41d1434d4604f9 2023-03-19 09:54:50,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:50,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:50,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:50,068 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:50,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:50,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:50,069 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,069 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,071 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:50,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:50,071 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 8a790cbed1774b39be41d1434d4604f9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:50,071 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:50,071 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=12, startTime=1679219690017; duration=0sec 2023-03-19 09:54:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:50,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,071 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:50,072 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:50,073 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45603 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:50,073 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:54:50,073 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,074 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/e5ec305e09f04f439d805d9d56b525a4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6e339cb2942b4939b625c1318d48471f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d808f211a6a2405d9be1311f78f14ada, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f3200995ac784716aa909850a497d7c7] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=44.5 K 2023-03-19 09:54:50,074 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e5ec305e09f04f439d805d9d56b525a4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679219686518 2023-03-19 09:54:50,074 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6e339cb2942b4939b625c1318d48471f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679219686550 2023-03-19 09:54:50,075 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d808f211a6a2405d9be1311f78f14ada, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1679219688678 2023-03-19 09:54:50,075 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f3200995ac784716aa909850a497d7c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219689124 2023-03-19 09:54:50,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:50,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:50,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:50,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:50,084 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:54:50,085 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:50,085 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:50,085 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:50,085 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:50,085 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:50,085 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:50,086 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:50,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:50,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219690264, max timestamp=1679219690278Num uniques 859; 2023-03-19 09:54:50,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:50,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219690264, max timestamp=1679219690278Num uniques 859; 2023-03-19 09:54:50,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:50,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219690264, max timestamp=1679219690278Num uniques 859; 2023-03-19 09:54:50,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:50,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219750293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219750294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219750294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219750295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219750296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219750397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219750399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219750399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219750400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219750400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,450 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/000bad7d9bdc4b9f97823f54e686d644 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/000bad7d9bdc4b9f97823f54e686d644 2023-03-19 09:54:50,456 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 000bad7d9bdc4b9f97823f54e686d644(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:50,456 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:50,456 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=12, startTime=1679219690017; duration=0sec 2023-03-19 09:54:50,456 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:50,503 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/4f4818d2aa3f49d6bced9544e98fc298 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4f4818d2aa3f49d6bced9544e98fc298 2023-03-19 09:54:50,508 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/8a71b8debcd0427883a299240f466bf6 2023-03-19 09:54:50,513 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into 4f4818d2aa3f49d6bced9544e98fc298(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:50,513 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:50,513 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=12, startTime=1679219690017; duration=0sec 2023-03-19 09:54:50,514 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:50,558 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/83b23e2b930b4522a7d36d5b8b8346d7 2023-03-19 09:54:50,585 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/4bb86a9f62e04458803b1090572978ea 2023-03-19 09:54:50,589 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/8a71b8debcd0427883a299240f466bf6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8a71b8debcd0427883a299240f466bf6 2023-03-19 09:54:50,594 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8a71b8debcd0427883a299240f466bf6, entries=150, sequenceid=427, filesize=11.9 K 2023-03-19 09:54:50,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/83b23e2b930b4522a7d36d5b8b8346d7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/83b23e2b930b4522a7d36d5b8b8346d7 2023-03-19 09:54:50,599 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/83b23e2b930b4522a7d36d5b8b8346d7, entries=150, sequenceid=427, filesize=11.9 K 2023-03-19 09:54:50,599 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/4bb86a9f62e04458803b1090572978ea as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4bb86a9f62e04458803b1090572978ea 2023-03-19 09:54:50,603 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4bb86a9f62e04458803b1090572978ea, entries=150, sequenceid=427, filesize=11.9 K 2023-03-19 09:54:50,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219750601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,604 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8dad03d10fa8d87944de8392ea9480f0 in 520ms, sequenceid=427, compaction requested=false 2023-03-19 09:54:50,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:50,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,606 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:50,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-19 09:54:50,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:50,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:50,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:50,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:50,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:50,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c1bdfa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:50,609 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:50,609 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:50,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:50,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:50,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:50,613 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:50,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:50,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:50,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:50,613 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:50,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:50,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,617 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/dd41ca664cb84bf794cdd3f306431487 2023-03-19 09:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219750616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,620 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:50,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219750616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,620 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:50,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:50,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:50,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:50,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:50,620 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:50,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,621 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:50,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:50,621 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:50,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219750620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219750622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,721 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219750721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219750721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:50,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:50,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:50,726 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:50,726 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,726 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:50,726 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:50,726 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,727 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219750726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219750726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,729 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:50,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:50,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:50,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:50,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:50,731 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:50,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:50,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f74c3ed[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:50,738 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:50,739 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,740 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:50,742 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,742 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:50,745 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:50,746 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:50,746 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:50,746 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:50,746 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:50,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:50,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,751 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,751 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,751 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,751 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,751 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,752 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,752 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,751 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:50,752 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:50,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:50,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:50,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:50,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:50,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:50,752 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:50,752 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:50,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:50,852 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:50,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:50,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:50,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:50,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:50,855 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:50,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:50,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:50,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:50,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:50,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:50,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:50,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:50,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:50,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:50,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:50,870 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e6e822[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:50,872 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:50,872 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,874 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:50,874 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,874 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:50,875 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:50,878 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:50,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:50,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:50,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:50,878 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:50,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:50,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:50,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:50,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:50,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:50,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:50,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:50,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:50,889 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:50,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,889 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:50,889 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:50,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:50,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:50,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219750906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219750923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219750924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219750929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:50,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219750929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,989 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:50,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:50,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:50,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:50,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:50,993 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:50,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:50,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:50,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,995 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:50,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:50,995 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,995 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,995 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:50,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:50,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:50,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:50,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:50,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:50,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:51,000 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb931fd[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:51,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:51,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:51,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,004 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:51,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,005 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,005 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,005 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:51,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:51,007 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:51,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:51,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:51,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:51,008 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:51,009 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,009 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,009 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,010 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,013 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:51,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:51,013 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:51,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,014 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,014 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:51,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,014 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:51,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,014 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/a02038ca0b2b470e918eb8f8a6b4dcff 2023-03-19 09:54:51,114 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:51,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:51,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:51,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:51,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:51,117 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:51,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,118 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:51,118 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:51,118 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,118 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,119 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:51,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:51,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:51,123 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:51,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739a6dbd[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:51,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:51,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,126 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,126 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:51,126 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:51,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:51,126 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:51,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:51,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:51,131 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:51,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:51,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:51,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:51,131 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:51,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:51,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:51,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:51,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,137 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,138 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:51,138 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:51,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,138 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,138 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,138 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35938 deadline: 1679219751226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219751228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35918 deadline: 1679219751231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35896 deadline: 1679219751234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,238 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:51,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:51,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:51,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:51,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:51,241 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:51,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:51,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:51,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:51,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:51,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:51,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:51,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a18b388[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:51,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:51,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:51,255 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:51,259 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:51,259 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:51,259 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:51,259 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:51,259 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:51,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,262 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,263 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,267 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,268 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:51,268 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,268 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,368 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:51,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:51,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:51,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:51,371 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:51,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:51,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:51,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,376 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,376 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,377 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,377 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:51,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:51,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:51,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:51,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:51,380 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:51,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:51,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298cef11[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:51,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:51,383 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:51,387 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:51,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:51,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:51,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:51,387 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:51,389 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,389 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,389 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:51,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:51,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:51,394 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:51,394 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:51,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219751409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/62ca9158033e4d28aa1ff2ac808e8fce 2023-03-19 09:54:51,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/dd41ca664cb84bf794cdd3f306431487 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dd41ca664cb84bf794cdd3f306431487 2023-03-19 09:54:51,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dd41ca664cb84bf794cdd3f306431487, entries=150, sequenceid=465, filesize=11.9 K 2023-03-19 09:54:51,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/a02038ca0b2b470e918eb8f8a6b4dcff as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a02038ca0b2b470e918eb8f8a6b4dcff 2023-03-19 09:54:51,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a02038ca0b2b470e918eb8f8a6b4dcff, entries=150, sequenceid=465, filesize=11.9 K 2023-03-19 09:54:51,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/62ca9158033e4d28aa1ff2ac808e8fce as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/62ca9158033e4d28aa1ff2ac808e8fce 2023-03-19 09:54:51,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/62ca9158033e4d28aa1ff2ac808e8fce, entries=150, sequenceid=465, filesize=11.9 K 2023-03-19 09:54:51,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 8dad03d10fa8d87944de8392ea9480f0 in 853ms, sequenceid=465, compaction requested=true 2023-03-19 09:54:51,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:51,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:51,460 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:51,460 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:51,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:51,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:51,461 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:51,461 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:54:51,461 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,462 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/000bad7d9bdc4b9f97823f54e686d644, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8a71b8debcd0427883a299240f466bf6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dd41ca664cb84bf794cdd3f306431487] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=35.9 K 2023-03-19 09:54:51,462 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:51,462 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:54:51,462 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,462 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8a790cbed1774b39be41d1434d4604f9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/83b23e2b930b4522a7d36d5b8b8346d7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a02038ca0b2b470e918eb8f8a6b4dcff] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=35.9 K 2023-03-19 09:54:51,464 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 000bad7d9bdc4b9f97823f54e686d644, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219689124 2023-03-19 09:54:51,464 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8a790cbed1774b39be41d1434d4604f9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219689124 2023-03-19 09:54:51,464 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8a71b8debcd0427883a299240f466bf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679219689156 2023-03-19 09:54:51,465 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 83b23e2b930b4522a7d36d5b8b8346d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679219689156 2023-03-19 09:54:51,465 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting dd41ca664cb84bf794cdd3f306431487, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219690292 2023-03-19 09:54:51,465 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a02038ca0b2b470e918eb8f8a6b4dcff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219690292 2023-03-19 09:54:51,474 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:51,486 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:51,494 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:51,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:51,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:51,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:51,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:51,497 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:51,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:51,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:51,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:51,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:51,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:51,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:51,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:51,502 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,502 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:51,502 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:54:51,502 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:51,502 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:51,502 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:51,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:51,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:51,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:51,523 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/dc2660baa6e54533b904e6ece9f64a3e 2023-03-19 09:54:51,538 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2940cba211164ba391270663a37e6857 2023-03-19 09:54:51,566 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/966f0042494a4599b16d990243a00cb7 2023-03-19 09:54:51,573 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/dc2660baa6e54533b904e6ece9f64a3e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dc2660baa6e54533b904e6ece9f64a3e 2023-03-19 09:54:51,577 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dc2660baa6e54533b904e6ece9f64a3e, entries=150, sequenceid=476, filesize=11.9 K 2023-03-19 09:54:51,578 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2940cba211164ba391270663a37e6857 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2940cba211164ba391270663a37e6857 2023-03-19 09:54:51,582 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2940cba211164ba391270663a37e6857, entries=150, sequenceid=476, filesize=11.9 K 2023-03-19 09:54:51,583 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/966f0042494a4599b16d990243a00cb7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/966f0042494a4599b16d990243a00cb7 2023-03-19 09:54:51,590 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/966f0042494a4599b16d990243a00cb7, entries=150, sequenceid=476, filesize=11.9 K 2023-03-19 09:54:51,590 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 8dad03d10fa8d87944de8392ea9480f0 in 88ms, sequenceid=476, compaction requested=true 2023-03-19 09:54:51,591 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:51,591 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:51,593 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8d0504[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:51,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:51,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,598 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:51,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:51,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:51,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:51,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:51,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:51,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:51,606 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:51,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:51,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:51,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:51,606 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:51,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,617 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:51,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:51,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,618 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,618 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:51,718 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:51,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:51,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:51,722 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:51,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:51,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:51,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:51,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:51,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:51,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:51,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:51,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:51,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:51,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:51,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:51,728 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:51,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e5a74ed[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:51,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:51,731 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:51,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:51,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:51,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:51,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:51,741 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:51,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:51,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:51,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:51,741 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:51,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,748 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:54:51,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:51,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:51,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:51,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:51,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:51,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:51,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:51,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:51,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,754 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:51,755 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:51,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/7e74d6b6257e44029cf141633c34d1b1 2023-03-19 09:54:51,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219691760, max timestamp=1679219691815Num uniques 859; 2023-03-19 09:54:51,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679219691760, max timestamp=1679219691815Num uniques 859; 2023-03-19 09:54:51,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679219691760, max timestamp=1679219691815Num uniques 859; 2023-03-19 09:54:51,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:51,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-19 09:54:51,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:54:51,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219751831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219751833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219751834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219751835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,855 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:51,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:51,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:51,860 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:51,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:51,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:51,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,863 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,863 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,864 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,864 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:51,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:51,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:51,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:51,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:51,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:51,866 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,866 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:51,866 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:51,866 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:51,867 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:51,867 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,867 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:51,867 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:51,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:51,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6101045d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:51,874 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:51,874 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,875 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:51,875 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,875 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:51,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:51,881 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:51,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:51,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:51,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:51,881 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:51,883 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,883 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,883 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:51,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:51,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:51,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:51,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:51,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:51,888 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:51,888 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:51,889 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:51,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:51,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:51,913 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/6b2a20c90cdd431999f195222f5f2854 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6b2a20c90cdd431999f195222f5f2854 2023-03-19 09:54:51,919 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 6b2a20c90cdd431999f195222f5f2854(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:51,919 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:51,919 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=13, startTime=1679219691460; duration=0sec 2023-03-19 09:54:51,920 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:51,920 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:51,921 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:51,921 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:54:51,921 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:51,921 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4f4818d2aa3f49d6bced9544e98fc298, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4bb86a9f62e04458803b1090572978ea, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/62ca9158033e4d28aa1ff2ac808e8fce, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/966f0042494a4599b16d990243a00cb7] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=47.8 K 2023-03-19 09:54:51,922 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4f4818d2aa3f49d6bced9544e98fc298, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219689124 2023-03-19 09:54:51,923 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4bb86a9f62e04458803b1090572978ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679219689156 2023-03-19 09:54:51,924 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 62ca9158033e4d28aa1ff2ac808e8fce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219690292 2023-03-19 09:54:51,924 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/75c78666f49c42f5bc87a01087aadf46 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/75c78666f49c42f5bc87a01087aadf46 2023-03-19 09:54:51,924 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 966f0042494a4599b16d990243a00cb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679219690610 2023-03-19 09:54:51,929 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 75c78666f49c42f5bc87a01087aadf46(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:51,929 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:51,929 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=13, startTime=1679219691460; duration=0sec 2023-03-19 09:54:51,929 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:51,934 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#233 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:51,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219751936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219751939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219751939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219751940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:51,972 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/43ab04ed702e438baf40dd96e6fe1a23 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/43ab04ed702e438baf40dd96e6fe1a23 2023-03-19 09:54:51,978 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into 43ab04ed702e438baf40dd96e6fe1a23(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:51,978 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:51,978 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=12, startTime=1679219691460; duration=0sec 2023-03-19 09:54:51,978 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:51,989 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:51,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:52,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:52,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:52,005 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:52,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:52,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:52,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,008 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:52,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:52,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:52,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:52,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:52,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbb8b0b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:52,022 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:52,022 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,023 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:52,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:52,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:52,025 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:52,031 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:52,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:52,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:52,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:52,031 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:52,033 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,033 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,034 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,036 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,038 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:52,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:52,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,044 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,044 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:52,044 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:52,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:52,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,144 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:52,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:52,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219752143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219752143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219752144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:52,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219752145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:52,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:52,147 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:52,147 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,147 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:52,147 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:52,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,149 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,149 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,150 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,150 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:52,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:52,151 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,151 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:52,151 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:52,151 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:52,153 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59681edd[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:52,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:52,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:52,156 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,156 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:52,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:52,159 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:52,162 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:52,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:52,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:52,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:52,162 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:52,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,164 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,165 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,168 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:52,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,168 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:52,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,168 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:52,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/3b3fcff14d3346d1964b889b5bb3b9c7 2023-03-19 09:54:52,269 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:52,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:52,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:52,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:52,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:52,273 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:52,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:52,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:52,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,275 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:52,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:52,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:52,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:52,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,284 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:52,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3478e401[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:52,286 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:52,286 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:52,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:52,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:52,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:52,295 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:52,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:52,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:52,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:52,296 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:52,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,302 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,302 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:52,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:52,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:52,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,303 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:52,303 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:52,302 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,403 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:52,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:52,406 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:52,406 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,406 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:52,406 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:52,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,408 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:52,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:52,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:52,410 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,410 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:52,410 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:52,410 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:52,412 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:52,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db22d3f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:52,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:52,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,416 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:52,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:52,418 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:52,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219752418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:52,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:52,421 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:52,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:52,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:52,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:52,421 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:52,422 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,422 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,422 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,423 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,426 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:52,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,427 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:52,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:52,427 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219752446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219752446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219752448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219752448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,527 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:52,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:52,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:52,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:52,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:52,530 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:52,530 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,530 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:52,530 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:52,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,533 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,534 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,534 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:52,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:52,535 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,535 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:52,535 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:52,535 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:52,537 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:52,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e220651[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:52,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:52,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,540 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:52,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:52,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:52,540 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,540 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:52,541 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:52,541 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,541 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,541 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:52,544 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:52,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:52,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:52,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:52,544 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:52,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:52,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,549 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:52,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,553 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:52,553 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,553 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:52,553 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:52,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:52,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:52,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/2172e02e22b440759b0243695144afb1 2023-03-19 09:54:52,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/7e74d6b6257e44029cf141633c34d1b1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7e74d6b6257e44029cf141633c34d1b1 2023-03-19 09:54:52,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7e74d6b6257e44029cf141633c34d1b1, entries=200, sequenceid=489, filesize=14.3 K 2023-03-19 09:54:52,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/3b3fcff14d3346d1964b889b5bb3b9c7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/3b3fcff14d3346d1964b889b5bb3b9c7 2023-03-19 09:54:52,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/3b3fcff14d3346d1964b889b5bb3b9c7, entries=150, sequenceid=489, filesize=11.9 K 2023-03-19 09:54:52,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/2172e02e22b440759b0243695144afb1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/2172e02e22b440759b0243695144afb1 2023-03-19 09:54:52,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/2172e02e22b440759b0243695144afb1, entries=150, sequenceid=489, filesize=11.9 K 2023-03-19 09:54:52,605 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.13 KB/120966 for 8dad03d10fa8d87944de8392ea9480f0 in 857ms, sequenceid=489, compaction requested=true 2023-03-19 09:54:52,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:52,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:52,606 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:52,606 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:52,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:52,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:52,607 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:52,607 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:52,607 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:54:52,607 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,607 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6b2a20c90cdd431999f195222f5f2854, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2940cba211164ba391270663a37e6857, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/3b3fcff14d3346d1964b889b5bb3b9c7] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=36.0 K 2023-03-19 09:54:52,607 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:54:52,607 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,607 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/75c78666f49c42f5bc87a01087aadf46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dc2660baa6e54533b904e6ece9f64a3e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7e74d6b6257e44029cf141633c34d1b1] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=38.4 K 2023-03-19 09:54:52,608 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6b2a20c90cdd431999f195222f5f2854, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219690292 2023-03-19 09:54:52,608 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 75c78666f49c42f5bc87a01087aadf46, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219690292 2023-03-19 09:54:52,608 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2940cba211164ba391270663a37e6857, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679219690610 2023-03-19 09:54:52,608 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting dc2660baa6e54533b904e6ece9f64a3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679219690610 2023-03-19 09:54:52,609 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 3b3fcff14d3346d1964b889b5bb3b9c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679219691745 2023-03-19 09:54:52,609 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7e74d6b6257e44029cf141633c34d1b1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679219691742 2023-03-19 09:54:52,616 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:52,616 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#235 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:52,638 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/493d760fdbba468cb4f8c005914eb9cd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/493d760fdbba468cb4f8c005914eb9cd 2023-03-19 09:54:52,642 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 493d760fdbba468cb4f8c005914eb9cd(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:52,642 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:52,642 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=13, startTime=1679219692606; duration=0sec 2023-03-19 09:54:52,643 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:52,643 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-19 09:54:52,643 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:54:52,643 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:54:52,643 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. because compaction request was cancelled 2023-03-19 09:54:52,653 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:52,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:52,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:52,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:52,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:52,656 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:52,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:52,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:52,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,658 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:52,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:52,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:52,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:52,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:52,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:52,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:52,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:52,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:52,660 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:52,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:52,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:52,660 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=118.13 KB heapSize=334.46 KB 2023-03-19 09:54:52,660 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:52,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:52,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:52,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:52,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:52,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:54:52,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:52,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:52,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219752959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219752960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:52,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219752961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:52,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219752961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,026 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/1c5f30a01b2e4f949d77c6957cb34e50 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/1c5f30a01b2e4f949d77c6957cb34e50 2023-03-19 09:54:53,031 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 1c5f30a01b2e4f949d77c6957cb34e50(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:53,032 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:53,032 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=13, startTime=1679219692606; duration=0sec 2023-03-19 09:54:53,032 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:53,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219753062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219753065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219753066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219753066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,081 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd 2023-03-19 09:54:53,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219753267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219753267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219753270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219753271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,497 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/e18446c0bb09483c80e8ccde2801a6c8 2023-03-19 09:54:53,507 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/0040728685154f848bd064631c2ccf13 2023-03-19 09:54:53,511 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd 2023-03-19 09:54:53,515 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd, entries=150, sequenceid=527, filesize=11.9 K 2023-03-19 09:54:53,516 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/e18446c0bb09483c80e8ccde2801a6c8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/e18446c0bb09483c80e8ccde2801a6c8 2023-03-19 09:54:53,519 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/e18446c0bb09483c80e8ccde2801a6c8, entries=150, sequenceid=527, filesize=11.9 K 2023-03-19 09:54:53,520 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/0040728685154f848bd064631c2ccf13 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/0040728685154f848bd064631c2ccf13 2023-03-19 09:54:53,533 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/0040728685154f848bd064631c2ccf13, entries=150, sequenceid=527, filesize=11.9 K 2023-03-19 09:54:53,534 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.13 KB/120966, heapSize ~334.41 KB/342440, currentSize=60.38 KB/61830 for 8dad03d10fa8d87944de8392ea9480f0 in 874ms, sequenceid=527, compaction requested=true 2023-03-19 09:54:53,534 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:53,534 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:53,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:53,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:53,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:53,537 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37257bbe[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:53,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:53,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,541 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:53,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:53,543 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,543 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,543 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,544 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:53,546 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:53,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:53,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:53,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:53,547 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:53,556 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,556 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,556 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,557 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,558 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,561 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:53,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:53,561 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:53,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,561 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:53,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:53,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:53,562 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,562 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:53,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:53,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:53,571 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:54:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:53,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/19cfd70fd0c745f4b3cfbc838a008f4a 2023-03-19 09:54:53,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219753587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219753588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219753589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219753590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,662 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:53,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:53,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:53,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:53,665 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:53,665 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:53,665 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:53,665 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:53,666 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,666 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:53,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:53,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:53,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:53,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:53,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:53,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:53,675 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:53,676 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@448ec9b7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:53,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:53,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:53,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:53,682 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:53,685 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:53,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:53,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:53,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:53,685 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:53,686 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,686 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,686 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:53,687 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:53,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:53,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:53,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:53,692 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,692 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:53,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,692 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:53,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219753692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219753693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219753693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219753695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,793 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:53,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:53,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:53,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:53,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:53,796 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:53,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:53,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:53,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:53,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,798 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:53,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:53,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:53,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:53,799 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:53,799 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:53,800 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:53,800 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,803 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@819dfcc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:53,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:53,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,805 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:53,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:53,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,807 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:53,807 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,807 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:53,809 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:53,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:53,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:53,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:53,810 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:53,811 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,811 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,811 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:53,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:53,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:53,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:53,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:53,817 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:53,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,817 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:53,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219753894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219753897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219753897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:53,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219753898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,917 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:53,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:53,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:53,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:53,921 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:53,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:53,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:53,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:53,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,923 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:53,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:53,923 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,923 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,923 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:53,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:53,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:53,926 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:53,926 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:53,926 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:53,926 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:53,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:53,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:53,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:53,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:53,928 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,928 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:53,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b75010[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:53,930 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:53,930 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:53,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,936 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:53,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:53,938 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:53,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:53,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:53,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:53,938 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:53,940 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,940 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,940 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:53,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,941 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:53,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:53,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:53,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:53,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:53,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:53,952 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:53,952 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:53,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/d254c1353b0e4539bcf288dfd3cb4d8b 2023-03-19 09:54:54,052 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:54,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:54,056 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:54,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:54,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:54,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,057 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,059 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:54,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:54,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:54,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:54,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:54,063 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a5325f3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:54,065 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:54,065 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:54,066 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,066 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:54,071 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:54,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:54,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:54,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:54,071 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:54,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,075 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,078 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:54,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:54,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,078 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:54,078 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:54,180 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:54,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:54,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:54,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:54,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:54,183 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:54,183 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,183 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:54,183 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:54,184 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,184 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,185 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,185 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,185 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,185 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:54,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:54,187 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,187 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:54,187 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:54,187 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:54,189 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:54,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219754196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:54,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1822927f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:54,202 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:54,202 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219754201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219754202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219754202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,204 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:54,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:54,206 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:54,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:54,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:54,209 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:54,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:54,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:54,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:54,209 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:54,211 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,218 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:54,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,218 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:54,218 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,318 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:54,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:54,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:54,322 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:54,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:54,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:54,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:54,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:54,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:54,338 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:54,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e83d908[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:54,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,341 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:54,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:54,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:54,341 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,341 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:54,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:54,342 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:54,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:54,345 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:54,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:54,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:54,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:54,345 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:54,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:54,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,356 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,356 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:54,357 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:54,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/bd818daa7d194af9a63876df59e9cc8c 2023-03-19 09:54:54,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/19cfd70fd0c745f4b3cfbc838a008f4a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/19cfd70fd0c745f4b3cfbc838a008f4a 2023-03-19 09:54:54,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/19cfd70fd0c745f4b3cfbc838a008f4a, entries=200, sequenceid=541, filesize=14.3 K 2023-03-19 09:54:54,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/d254c1353b0e4539bcf288dfd3cb4d8b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/d254c1353b0e4539bcf288dfd3cb4d8b 2023-03-19 09:54:54,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/d254c1353b0e4539bcf288dfd3cb4d8b, entries=150, sequenceid=541, filesize=11.9 K 2023-03-19 09:54:54,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/bd818daa7d194af9a63876df59e9cc8c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/bd818daa7d194af9a63876df59e9cc8c 2023-03-19 09:54:54,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35930 deadline: 1679219754437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,441 DEBUG [Thread-1600] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:54:54,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/bd818daa7d194af9a63876df59e9cc8c, entries=150, sequenceid=541, filesize=11.9 K 2023-03-19 09:54:54,442 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8dad03d10fa8d87944de8392ea9480f0 in 871ms, sequenceid=541, compaction requested=true 2023-03-19 09:54:54,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:54,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:54,442 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:54,442 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:54,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:54,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:54,443 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:54,444 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:54:54,444 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,444 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/493d760fdbba468cb4f8c005914eb9cd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/e18446c0bb09483c80e8ccde2801a6c8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/d254c1353b0e4539bcf288dfd3cb4d8b] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=36.1 K 2023-03-19 09:54:54,444 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:54,444 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:54:54,444 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 493d760fdbba468cb4f8c005914eb9cd, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679219691745 2023-03-19 09:54:54,444 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,445 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/1c5f30a01b2e4f949d77c6957cb34e50, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/19cfd70fd0c745f4b3cfbc838a008f4a] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=38.5 K 2023-03-19 09:54:54,445 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e18446c0bb09483c80e8ccde2801a6c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219691827 2023-03-19 09:54:54,445 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1c5f30a01b2e4f949d77c6957cb34e50, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679219691745 2023-03-19 09:54:54,445 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d254c1353b0e4539bcf288dfd3cb4d8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1679219692959 2023-03-19 09:54:54,445 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 18fe04bb8d1749b6ab7b2c4ddf63ffbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219691827 2023-03-19 09:54:54,446 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 19cfd70fd0c745f4b3cfbc838a008f4a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1679219692955 2023-03-19 09:54:54,456 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:54,457 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:54,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:54,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:54,461 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:54,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:54,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:54,462 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:54,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:54,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:54,463 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,463 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,467 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:54,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:54,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:54,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:54,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:54,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:54,479 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/1e4dc07390184322abd04fdc80452627 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e4dc07390184322abd04fdc80452627 2023-03-19 09:54:54,483 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 1e4dc07390184322abd04fdc80452627(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:54,484 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:54,484 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=13, startTime=1679219694442; duration=0sec 2023-03-19 09:54:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:54,484 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:54,484 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:54,485 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:54,485 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:54:54,485 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,485 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/43ab04ed702e438baf40dd96e6fe1a23, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/2172e02e22b440759b0243695144afb1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/0040728685154f848bd064631c2ccf13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/bd818daa7d194af9a63876df59e9cc8c] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=48.0 K 2023-03-19 09:54:54,486 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 43ab04ed702e438baf40dd96e6fe1a23, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679219690610 2023-03-19 09:54:54,486 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2172e02e22b440759b0243695144afb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679219691745 2023-03-19 09:54:54,486 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0040728685154f848bd064631c2ccf13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219691827 2023-03-19 09:54:54,487 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting bd818daa7d194af9a63876df59e9cc8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1679219692959 2023-03-19 09:54:54,488 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,488 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:54,488 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-19 09:54:54,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:54,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:54,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:54,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:54,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:54,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:54,498 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:54,514 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/5d3f043ed2b242bbbe2c9ec1adc09d9b 2023-03-19 09:54:54,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:54,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:54,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219754712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219754712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219754713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219754714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219754815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219754815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219754816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:54,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219754818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,875 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/9d96839610e44f389a9cf942a3f8bdb3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d96839610e44f389a9cf942a3f8bdb3 2023-03-19 09:54:54,879 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 9d96839610e44f389a9cf942a3f8bdb3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:54,880 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:54,880 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=13, startTime=1679219694442; duration=0sec 2023-03-19 09:54:54,880 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:54,920 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/cd9e5ac6ee6a4c228f06025633f2817e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/cd9e5ac6ee6a4c228f06025633f2817e 2023-03-19 09:54:54,925 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into cd9e5ac6ee6a4c228f06025633f2817e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:54,925 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:54,925 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=12, startTime=1679219694443; duration=0sec 2023-03-19 09:54:54,925 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:54,925 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/1f7f03f4dc3d437aa4e8a69c45104a28 2023-03-19 09:54:54,954 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/fe668b971d624c4db614f92e7ebcac46 2023-03-19 09:54:54,958 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/5d3f043ed2b242bbbe2c9ec1adc09d9b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5d3f043ed2b242bbbe2c9ec1adc09d9b 2023-03-19 09:54:54,962 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5d3f043ed2b242bbbe2c9ec1adc09d9b, entries=150, sequenceid=561, filesize=11.9 K 2023-03-19 09:54:54,963 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/1f7f03f4dc3d437aa4e8a69c45104a28 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1f7f03f4dc3d437aa4e8a69c45104a28 2023-03-19 09:54:54,967 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1f7f03f4dc3d437aa4e8a69c45104a28, entries=150, sequenceid=561, filesize=11.9 K 2023-03-19 09:54:54,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/fe668b971d624c4db614f92e7ebcac46 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe668b971d624c4db614f92e7ebcac46 2023-03-19 09:54:54,973 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe668b971d624c4db614f92e7ebcac46, entries=150, sequenceid=561, filesize=11.9 K 2023-03-19 09:54:54,974 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8dad03d10fa8d87944de8392ea9480f0 in 486ms, sequenceid=561, compaction requested=false 2023-03-19 09:54:54,974 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:54,974 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:54,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:54,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52382518[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:54,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:54,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,979 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:54,979 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:54,979 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,979 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:54,981 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,981 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:54,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:54,984 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:54,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:54,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:54,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:54,984 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:54,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:54,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:54,987 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:54,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:54,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:54,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:54,990 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:54,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:54,990 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:54,990 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:54,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:54,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:55,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:54:55,021 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:55,021 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:55,021 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:55,021 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:55,021 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:55,021 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:55,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219755032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219755033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219755034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219755035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,090 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:55,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:55,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:55,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:55,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:55,094 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:55,094 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,094 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:55,094 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:55,094 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,094 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,095 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:55,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:55,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:55,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:55,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:55,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:55,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:55,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:55,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:55,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:55,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:55,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:55,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:55,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:55,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:55,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f7ee11[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:55,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,103 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:55,103 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,103 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,104 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:55,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:55,107 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:55,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:55,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:55,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:55,107 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:55,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,114 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:55,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,114 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:55,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,115 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,115 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,115 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,115 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:55,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219755136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219755138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219755139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219755139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,215 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:55,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:55,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:55,220 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:55,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:55,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:55,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,222 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:55,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:55,230 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:55,231 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,231 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:55,231 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:55,231 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:55,233 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,234 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:55,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dca293a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:55,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:55,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:55,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:55,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:55,243 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:55,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:55,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:55,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:55,243 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:55,245 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,245 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,245 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,247 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:55,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:55,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,250 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:55,251 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:55,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219755339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219755339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219755342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219755343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,351 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:55,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:55,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:55,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:55,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:55,354 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:55,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:55,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:55,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,356 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,356 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,356 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:55,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:55,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:55,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:55,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:55,358 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:55,358 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:55,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:55,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d41f612[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:55,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:55,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:55,368 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:55,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:55,370 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:55,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:55,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:55,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:55,370 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:55,373 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,373 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,373 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:55,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,378 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:55,378 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:55,378 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/95d060f48d7547ce8d9e6876416c55b8 2023-03-19 09:54:55,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/16e42ca82ba644acbfa1e4c9a9e87541 2023-03-19 09:54:55,478 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:55,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:55,481 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:55,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:55,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:55,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,483 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:55,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:55,485 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,485 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:55,485 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:55,485 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:55,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,486 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec21647[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:55,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:55,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:55,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:55,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:55,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:55,492 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:55,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:55,495 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:55,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:55,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:55,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:55,495 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:55,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:55,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:55,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,501 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:55,501 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:55,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,602 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:55,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:55,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:55,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:55,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:55,606 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:55,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:55,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:55,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,609 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,609 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,609 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,609 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:55,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:55,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:55,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:55,618 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,620 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:55,620 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:55,620 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,622 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601bf0e8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:55,625 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:55,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:55,628 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,628 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:55,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:55,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:55,629 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:55,633 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:55,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:55,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:55,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:55,633 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:55,636 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,636 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,636 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,637 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,642 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:55,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:55,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,643 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,643 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,643 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:55,643 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:55,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:55,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219755642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219755642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219755646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219755647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,743 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:55,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:55,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:55,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:55,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:55,747 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:55,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:55,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:55,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:55,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:55,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:55,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:55,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:55,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:55,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:55,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:55,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca48628[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:55,756 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:55,756 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:55,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,759 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:55,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,763 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,763 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,763 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:55,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:55,766 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:55,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:55,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:55,766 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:55,766 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:55,768 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,768 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,768 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,769 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,770 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,772 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,773 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,773 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,773 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,773 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,773 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,773 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,772 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:55,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:55,773 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,773 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:55,773 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:55,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:55,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c0fe671e9e4b4bddaf2a63e02d8ab8af 2023-03-19 09:54:55,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/95d060f48d7547ce8d9e6876416c55b8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/95d060f48d7547ce8d9e6876416c55b8 2023-03-19 09:54:55,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/95d060f48d7547ce8d9e6876416c55b8, entries=200, sequenceid=579, filesize=14.3 K 2023-03-19 09:54:55,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/16e42ca82ba644acbfa1e4c9a9e87541 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/16e42ca82ba644acbfa1e4c9a9e87541 2023-03-19 09:54:55,873 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:55,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:55,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:55,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:55,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:55,877 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:55,877 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,877 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:55,877 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:55,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/16e42ca82ba644acbfa1e4c9a9e87541, entries=150, sequenceid=579, filesize=11.9 K 2023-03-19 09:54:55,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c0fe671e9e4b4bddaf2a63e02d8ab8af as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c0fe671e9e4b4bddaf2a63e02d8ab8af 2023-03-19 09:54:55,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,882 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,882 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:55,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:55,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:55,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:55,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:55,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:55,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:55,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:55,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:55,887 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:55,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:55,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:55,889 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c0fe671e9e4b4bddaf2a63e02d8ab8af, entries=150, sequenceid=579, filesize=11.9 K 2023-03-19 09:54:55,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,890 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,890 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8dad03d10fa8d87944de8392ea9480f0 in 869ms, sequenceid=579, compaction requested=true 2023-03-19 09:54:55,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:55,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:55,891 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:55,891 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:55,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,896 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:55,896 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:54:55,896 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,896 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d96839610e44f389a9cf942a3f8bdb3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5d3f043ed2b242bbbe2c9ec1adc09d9b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/95d060f48d7547ce8d9e6876416c55b8] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=38.6 K 2023-03-19 09:54:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,897 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:55,897 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:54:55,897 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,897 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e4dc07390184322abd04fdc80452627, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1f7f03f4dc3d437aa4e8a69c45104a28, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/16e42ca82ba644acbfa1e4c9a9e87541] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=36.2 K 2023-03-19 09:54:55,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:55,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef71ac8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:55,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:55,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,897 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9d96839610e44f389a9cf942a3f8bdb3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1679219692959 2023-03-19 09:54:55,898 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1e4dc07390184322abd04fdc80452627, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1679219692959 2023-03-19 09:54:55,898 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1f7f03f4dc3d437aa4e8a69c45104a28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1679219693586 2023-03-19 09:54:55,898 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5d3f043ed2b242bbbe2c9ec1adc09d9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1679219693586 2023-03-19 09:54:55,898 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 16e42ca82ba644acbfa1e4c9a9e87541, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679219694712 2023-03-19 09:54:55,899 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 95d060f48d7547ce8d9e6876416c55b8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679219694712 2023-03-19 09:54:55,899 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:55,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:55,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:55,901 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:55,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:55,907 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:55,907 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:55,907 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:55,907 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:55,907 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:55,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:55,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:55,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,910 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:55,912 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:55,915 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:55,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:55,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:55,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:55,916 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:55,916 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:55,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:55,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:55,921 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:55,925 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:55,965 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/a76afd45dd2c413e95d32aeaa21e0471 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a76afd45dd2c413e95d32aeaa21e0471 2023-03-19 09:54:55,976 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into a76afd45dd2c413e95d32aeaa21e0471(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:55,976 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:55,976 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=13, startTime=1679219695891; duration=0sec 2023-03-19 09:54:55,976 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:55,977 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:54:55,978 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:54:55,978 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:54:55,978 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:55,979 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/cd9e5ac6ee6a4c228f06025633f2817e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe668b971d624c4db614f92e7ebcac46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c0fe671e9e4b4bddaf2a63e02d8ab8af] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=36.2 K 2023-03-19 09:54:55,979 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting cd9e5ac6ee6a4c228f06025633f2817e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1679219692959 2023-03-19 09:54:55,980 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fe668b971d624c4db614f92e7ebcac46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1679219693586 2023-03-19 09:54:55,984 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c0fe671e9e4b4bddaf2a63e02d8ab8af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679219694712 2023-03-19 09:54:55,994 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/5cbdfae384c14cdb8f3e7e38309c74d0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5cbdfae384c14cdb8f3e7e38309c74d0 2023-03-19 09:54:56,001 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:56,012 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 5cbdfae384c14cdb8f3e7e38309c74d0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:56,012 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:56,012 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=13, startTime=1679219695890; duration=0sec 2023-03-19 09:54:56,012 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:56,016 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:56,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:56,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:56,021 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:56,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:56,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:56,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:56,022 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,022 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:56,024 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:56,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:56,025 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,025 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:56,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:56,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:56,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:56,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:56,036 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:56,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:56,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:56,041 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:56,041 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:56,041 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:54:56,041 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:56,041 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:56,041 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:56,041 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:56,041 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:56,042 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:56,045 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/b0cee81b47f14edbbf50e3cdf5515fcc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b0cee81b47f14edbbf50e3cdf5515fcc 2023-03-19 09:54:56,051 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into b0cee81b47f14edbbf50e3cdf5515fcc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:56,051 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:56,051 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=13, startTime=1679219695891; duration=0sec 2023-03-19 09:54:56,051 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:56,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:56,147 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:56,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219756160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219756159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219756159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219756160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219756264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219756264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219756265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219756265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,454 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/f602f3de205f408fb8b59b647b97a57f 2023-03-19 09:54:56,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219756467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219756468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219756468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219756468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219756770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219756772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219756772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219756773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,865 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/346ec05d7c7443328b6400f2ef376a82 2023-03-19 09:54:56,878 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/b5ab37b6820c44e390121d5247ee3ff3 2023-03-19 09:54:56,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/f602f3de205f408fb8b59b647b97a57f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f602f3de205f408fb8b59b647b97a57f 2023-03-19 09:54:56,895 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f602f3de205f408fb8b59b647b97a57f, entries=150, sequenceid=597, filesize=11.9 K 2023-03-19 09:54:56,896 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/346ec05d7c7443328b6400f2ef376a82 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/346ec05d7c7443328b6400f2ef376a82 2023-03-19 09:54:56,901 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/346ec05d7c7443328b6400f2ef376a82, entries=150, sequenceid=597, filesize=11.9 K 2023-03-19 09:54:56,902 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/b5ab37b6820c44e390121d5247ee3ff3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b5ab37b6820c44e390121d5247ee3ff3 2023-03-19 09:54:56,907 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b5ab37b6820c44e390121d5247ee3ff3, entries=150, sequenceid=597, filesize=11.9 K 2023-03-19 09:54:56,908 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 8dad03d10fa8d87944de8392ea9480f0 in 867ms, sequenceid=597, compaction requested=false 2023-03-19 09:54:56,908 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:56,908 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a95b7f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:56,912 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:56,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,914 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:56,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:56,915 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,915 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:56,916 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:56,918 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:56,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:56,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:56,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:56,918 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:56,920 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,920 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,920 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:56,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:56,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:56,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,923 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:56,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:56,926 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:56,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:56,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:56,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:56,927 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:56,927 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:56,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,027 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:57,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:57,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:57,030 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:57,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:57,030 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:57,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,033 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:57,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:57,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:57,035 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,035 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:57,035 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:54:57,035 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:57,044 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/b364d9ece9e34316998c17732d0f3b8d 2023-03-19 09:54:57,055 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/6402ccc8cc51453f8335685efb73602f 2023-03-19 09:54:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:57,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:57,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219757284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219757286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219757287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219757287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219757388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219757388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219757390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219757390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,467 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/db8ed20c5be944ff9bd3e85a28f59c83 2023-03-19 09:54:57,471 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/b364d9ece9e34316998c17732d0f3b8d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b364d9ece9e34316998c17732d0f3b8d 2023-03-19 09:54:57,475 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b364d9ece9e34316998c17732d0f3b8d, entries=150, sequenceid=615, filesize=11.9 K 2023-03-19 09:54:57,475 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/6402ccc8cc51453f8335685efb73602f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6402ccc8cc51453f8335685efb73602f 2023-03-19 09:54:57,479 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6402ccc8cc51453f8335685efb73602f, entries=150, sequenceid=615, filesize=11.9 K 2023-03-19 09:54:57,479 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/db8ed20c5be944ff9bd3e85a28f59c83 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/db8ed20c5be944ff9bd3e85a28f59c83 2023-03-19 09:54:57,483 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/db8ed20c5be944ff9bd3e85a28f59c83, entries=150, sequenceid=615, filesize=11.9 K 2023-03-19 09:54:57,483 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8dad03d10fa8d87944de8392ea9480f0 in 448ms, sequenceid=615, compaction requested=true 2023-03-19 09:54:57,483 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:57,483 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,483 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:57,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:57,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:57,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:57,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,486 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,486 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8a6d32[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:57,487 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:57,487 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,488 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:57,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:57,490 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:57,490 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,490 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,490 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:57,492 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:57,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:57,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:57,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:57,492 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:57,498 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,498 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,498 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:57,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:57,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,507 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,507 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:57,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:54:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:57,607 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:57,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:57,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219757605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:57,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219757606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219757607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:57,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219757608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,610 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:57,610 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,610 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:57,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:57,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,612 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:57,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:57,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:57,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:57,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:57,614 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,614 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:57,614 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:57,614 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:57,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:57,617 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:57,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1767332d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:57,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:57,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,620 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:57,620 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,620 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:57,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:57,622 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:57,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:57,624 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:57,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:57,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:57,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:57,624 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:57,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:57,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:57,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,630 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,630 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,631 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:57,631 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:57,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219757709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219757711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219757712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219757712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,731 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:57,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:57,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:57,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:57,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:57,734 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:57,734 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,734 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:57,734 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:57,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:57,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:57,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:57,737 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:57,738 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,738 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:57,738 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:57,738 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:57,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:57,740 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,740 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,740 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,740 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:57,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51004a53[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:57,742 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:57,742 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:57,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:57,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:57,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:57,746 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:57,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:57,748 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:57,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:57,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:57,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:57,748 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:57,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:57,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:57,754 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:57,754 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,854 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:57,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:57,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:57,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:57,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:57,857 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:57,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:57,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:57,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,859 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:57,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:57,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:57,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:57,862 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:57,864 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be891af[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:57,866 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:57,866 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,867 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:57,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:57,869 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:57,871 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:57,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:57,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:57,872 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:57,872 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:57,873 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,873 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,873 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,874 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,875 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:57,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:57,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:57,881 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:57,881 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:57,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:57,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,881 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219757911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219757913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219757915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219757916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,981 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:57,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:57,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:57,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:57,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:57,984 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:57,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:57,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:57,985 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,985 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:57,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:57,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:57,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:57,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:57,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:57,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:57,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:57,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:57,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:57,990 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:57,990 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:57,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,992 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,992 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,992 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:57,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7737961d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:57,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:57,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,994 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:57,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,995 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:57,997 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:57,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:57,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:57,999 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:57,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:57,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:57,999 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:57,999 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,001 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,001 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,001 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,001 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,005 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,005 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:58,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/c5d865bf5a83489a8cd85475fe4935e4 2023-03-19 09:54:58,105 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:58,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:58,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:58,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:58,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:58,109 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:58,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:58,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:58,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:58,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:58,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:58,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:58,114 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,114 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:58,114 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:58,114 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:58,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,117 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e72d8d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:58,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:58,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,120 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:58,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:58,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:58,124 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:58,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:58,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:58,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:58,124 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,126 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,126 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,126 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,131 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,131 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:58,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219758215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219758218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219758220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219758221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,231 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:58,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:58,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:58,235 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:58,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:58,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:58,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:58,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:58,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:58,239 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,239 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:58,239 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:58,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:58,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:58,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae6b630[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:58,243 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,244 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:58,245 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,245 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:58,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:58,252 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:58,256 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:58,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:58,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:58,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:58,256 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,258 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,258 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,258 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,261 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,265 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,265 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,366 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:58,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:58,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:58,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:58,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:58,369 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:58,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:58,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:58,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:58,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:58,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:58,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:58,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:58,376 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:58,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c2b2a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:58,379 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:58,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,381 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:58,381 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:58,381 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:58,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:58,382 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:58,384 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:58,385 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:58,385 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:58,385 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:58,385 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,387 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,387 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,387 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,391 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,391 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/c43c385127434c9f93fba47e66cf4f8f 2023-03-19 09:54:58,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/114148a181854621bfe5e6dc4d95e4a0 2023-03-19 09:54:58,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/c5d865bf5a83489a8cd85475fe4935e4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c5d865bf5a83489a8cd85475fe4935e4 2023-03-19 09:54:58,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c5d865bf5a83489a8cd85475fe4935e4, entries=150, sequenceid=632, filesize=11.9 K 2023-03-19 09:54:58,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/c43c385127434c9f93fba47e66cf4f8f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c43c385127434c9f93fba47e66cf4f8f 2023-03-19 09:54:58,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c43c385127434c9f93fba47e66cf4f8f, entries=150, sequenceid=632, filesize=11.9 K 2023-03-19 09:54:58,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/114148a181854621bfe5e6dc4d95e4a0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/114148a181854621bfe5e6dc4d95e4a0 2023-03-19 09:54:58,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/114148a181854621bfe5e6dc4d95e4a0, entries=150, sequenceid=632, filesize=11.9 K 2023-03-19 09:54:58,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 8dad03d10fa8d87944de8392ea9480f0 in 858ms, sequenceid=632, compaction requested=true 2023-03-19 09:54:58,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:58,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:58,451 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:58,451 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:58,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:58,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:58,454 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:58,454 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:54:58,454 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,454 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5cbdfae384c14cdb8f3e7e38309c74d0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f602f3de205f408fb8b59b647b97a57f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b364d9ece9e34316998c17732d0f3b8d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c5d865bf5a83489a8cd85475fe4935e4] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=48.2 K 2023-03-19 09:54:58,455 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:58,455 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:54:58,455 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,455 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5cbdfae384c14cdb8f3e7e38309c74d0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679219694712 2023-03-19 09:54:58,455 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a76afd45dd2c413e95d32aeaa21e0471, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/346ec05d7c7443328b6400f2ef376a82, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6402ccc8cc51453f8335685efb73602f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c43c385127434c9f93fba47e66cf4f8f] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=48.2 K 2023-03-19 09:54:58,455 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting f602f3de205f408fb8b59b647b97a57f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1679219695032 2023-03-19 09:54:58,456 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a76afd45dd2c413e95d32aeaa21e0471, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679219694712 2023-03-19 09:54:58,456 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b364d9ece9e34316998c17732d0f3b8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1679219696158 2023-03-19 09:54:58,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:58,457 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 346ec05d7c7443328b6400f2ef376a82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1679219695032 2023-03-19 09:54:58,457 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c5d865bf5a83489a8cd85475fe4935e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679219697281 2023-03-19 09:54:58,457 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6402ccc8cc51453f8335685efb73602f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1679219696158 2023-03-19 09:54:58,457 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:54:58,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:58,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:58,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:58,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:58,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:58,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:58,458 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c43c385127434c9f93fba47e66cf4f8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679219697281 2023-03-19 09:54:58,474 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:58,478 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#266 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:58,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/d225887b4d9e45dc956959d255032554 2023-03-19 09:54:58,491 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:58,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:58,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:58,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:58,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:58,497 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:58,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:58,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:58,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,498 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/8b9a6fd8c5d24108aea96098f105dbba as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8b9a6fd8c5d24108aea96098f105dbba 2023-03-19 09:54:58,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:58,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:58,502 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 8b9a6fd8c5d24108aea96098f105dbba(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:58,502 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:58,502 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=12, startTime=1679219698451; duration=0sec 2023-03-19 09:54:58,502 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:54:58,502 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:54:58,503 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:54:58,503 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:54:58,504 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,504 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b0cee81b47f14edbbf50e3cdf5515fcc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b5ab37b6820c44e390121d5247ee3ff3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/db8ed20c5be944ff9bd3e85a28f59c83, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/114148a181854621bfe5e6dc4d95e4a0] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=48.2 K 2023-03-19 09:54:58,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:58,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:58,504 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b0cee81b47f14edbbf50e3cdf5515fcc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679219694712 2023-03-19 09:54:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:58,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:58,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:58,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,505 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b5ab37b6820c44e390121d5247ee3ff3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1679219695032 2023-03-19 09:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:58,505 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting db8ed20c5be944ff9bd3e85a28f59c83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1679219696158 2023-03-19 09:54:58,505 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 114148a181854621bfe5e6dc4d95e4a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679219697281 2023-03-19 09:54:58,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219758505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:58,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec0d2e8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:58,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:58,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:58,514 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#268 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:54:58,520 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:58,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:58,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:58,521 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:58,521 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,521 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:58,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:58,524 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:58,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:58,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:58,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:58,524 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,527 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,531 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,532 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,532 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,535 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/9ad49f3562f14d3680575ef590d7f7b1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/9ad49f3562f14d3680575ef590d7f7b1 2023-03-19 09:54:58,541 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into 9ad49f3562f14d3680575ef590d7f7b1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:58,541 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:58,541 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=12, startTime=1679219698453; duration=0sec 2023-03-19 09:54:58,541 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:58,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219758610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,632 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:58,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:58,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:58,635 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:58,635 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,635 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:58,635 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:58,636 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,636 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,638 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:58,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:58,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:58,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:58,640 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,641 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:58,641 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:58,641 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:58,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:58,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eab15f8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:58,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,645 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:58,646 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,646 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:58,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:58,650 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:58,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:58,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:58,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:58,650 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,652 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,652 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,652 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,653 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,654 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,656 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,656 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,656 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,656 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,656 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,659 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,659 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219758718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219758723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219758724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219758725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,759 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:58,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:58,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:58,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:58,763 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:58,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:58,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:58,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,764 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,764 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:58,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:58,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:58,766 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,766 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:58,767 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:58,772 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,772 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1143612b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:58,774 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:58,774 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,775 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,775 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:58,775 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:58,775 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:58,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:58,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:58,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:58,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:58,779 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:58,779 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:58,779 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:58,779 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:58,779 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,782 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,789 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,789 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,789 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,789 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,789 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,790 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,790 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,790 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,790 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,790 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,790 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219758813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,890 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:58,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:58,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:58,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:58,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:58,895 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:58,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:58,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:58,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,896 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:58,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:58,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:58,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:58,898 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:58,898 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:58,898 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:58,898 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:58,898 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:58,898 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,899 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:58,899 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:58,899 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:58,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:58,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/1479a193bdd147f5bc446b2e683e1de5 2023-03-19 09:54:58,900 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/94edf94645b34afa9ad95a1bf748a9a2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/94edf94645b34afa9ad95a1bf748a9a2 2023-03-19 09:54:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,901 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:58,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75724d52[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:58,907 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:58,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:58,912 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:58,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:58,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:58,913 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:58,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:58,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:58,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:58,913 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:58,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,916 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,916 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:58,916 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:58,916 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 94edf94645b34afa9ad95a1bf748a9a2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:54:58,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,918 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:58,918 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=12, startTime=1679219698451; duration=0sec 2023-03-19 09:54:58,918 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:54:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:58,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,922 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:58,922 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:58,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:58,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/6c55f107d33949f38e49f511c79036c4 2023-03-19 09:54:58,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:58,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:58,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:58,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,929 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:58,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:58,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:58,930 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:58,930 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:58,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:58,930 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:58,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:54:58,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/d225887b4d9e45dc956959d255032554 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/d225887b4d9e45dc956959d255032554 2023-03-19 09:54:58,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/d225887b4d9e45dc956959d255032554, entries=150, sequenceid=649, filesize=11.9 K 2023-03-19 09:54:58,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/1479a193bdd147f5bc446b2e683e1de5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1479a193bdd147f5bc446b2e683e1de5 2023-03-19 09:54:58,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1479a193bdd147f5bc446b2e683e1de5, entries=150, sequenceid=649, filesize=11.9 K 2023-03-19 09:54:58,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/6c55f107d33949f38e49f511c79036c4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6c55f107d33949f38e49f511c79036c4 2023-03-19 09:54:58,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6c55f107d33949f38e49f511c79036c4, entries=150, sequenceid=649, filesize=11.9 K 2023-03-19 09:54:58,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8dad03d10fa8d87944de8392ea9480f0 in 494ms, sequenceid=649, compaction requested=false 2023-03-19 09:54:58,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:59,030 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:59,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:59,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:59,033 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:59,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:59,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:59,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:59,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:59,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:59,036 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,036 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:59,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:59,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:59,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:59,037 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:59,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:59,038 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:54:59,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:59,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:59,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:59,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:59,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:59,038 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:59,054 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/516c084a6cbe4a0f8a4c4c424a2f9800 2023-03-19 09:54:59,073 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/babd198595a44f53b42a4c2ee94acc70 2023-03-19 09:54:59,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:59,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:59,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219759160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219759263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219759468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,507 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/66559e4f72e54e00ae8ed47cdf5b5769 2023-03-19 09:54:59,511 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/516c084a6cbe4a0f8a4c4c424a2f9800 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/516c084a6cbe4a0f8a4c4c424a2f9800 2023-03-19 09:54:59,515 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/516c084a6cbe4a0f8a4c4c424a2f9800, entries=150, sequenceid=667, filesize=11.9 K 2023-03-19 09:54:59,516 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/babd198595a44f53b42a4c2ee94acc70 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/babd198595a44f53b42a4c2ee94acc70 2023-03-19 09:54:59,523 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/babd198595a44f53b42a4c2ee94acc70, entries=150, sequenceid=667, filesize=11.9 K 2023-03-19 09:54:59,524 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/66559e4f72e54e00ae8ed47cdf5b5769 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/66559e4f72e54e00ae8ed47cdf5b5769 2023-03-19 09:54:59,529 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/66559e4f72e54e00ae8ed47cdf5b5769, entries=150, sequenceid=667, filesize=11.9 K 2023-03-19 09:54:59,530 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8dad03d10fa8d87944de8392ea9480f0 in 492ms, sequenceid=667, compaction requested=true 2023-03-19 09:54:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:54:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:54:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:54:59,542 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d6d8686[Count = 0] remaining members to acquire global barrier 2023-03-19 09:54:59,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:54:59,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,545 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,546 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:54:59,546 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:54:59,546 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:54:59,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:54:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:54:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:59,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:54:59,553 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:54:59,553 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:54:59,553 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:54:59,553 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:54:59,553 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:54:59,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:59,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:54:59,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:54:59,558 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:54:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,565 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:54:59,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:54:59,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:54:59,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:54:59,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,566 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:54:59,566 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:54:59,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:54:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:54:59,666 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:54:59,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:54:59,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:54:59,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:54:59,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:54:59,669 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:54:59,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:59,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:54:59,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:54:59,670 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,670 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,671 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:54:59,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:54:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:54:59,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:54:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:54:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:54:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:54:59,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:54:59,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:54:59,673 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:54:59,673 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:54:59,673 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:54:59,673 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:54:59,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:59,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:54:59,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:59,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:54:59,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:54:59,686 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/589d53e2fc81420a81836eacfb5b8a53 2023-03-19 09:54:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:54:59,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:54:59,742 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5 2023-03-19 09:54:59,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219759746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219759746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219759750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219759751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219759773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219759852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219759852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219759854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:54:59,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:54:59,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219759854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219760055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219760056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219760057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219760059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,155 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c6541b8544ff4b42970aca65a15c50c7 2023-03-19 09:55:00,159 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/589d53e2fc81420a81836eacfb5b8a53 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/589d53e2fc81420a81836eacfb5b8a53 2023-03-19 09:55:00,162 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/589d53e2fc81420a81836eacfb5b8a53, entries=150, sequenceid=684, filesize=11.9 K 2023-03-19 09:55:00,163 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5 2023-03-19 09:55:00,166 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5, entries=150, sequenceid=684, filesize=11.9 K 2023-03-19 09:55:00,167 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c6541b8544ff4b42970aca65a15c50c7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c6541b8544ff4b42970aca65a15c50c7 2023-03-19 09:55:00,171 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c6541b8544ff4b42970aca65a15c50c7, entries=150, sequenceid=684, filesize=11.9 K 2023-03-19 09:55:00,172 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 8dad03d10fa8d87944de8392ea9480f0 in 499ms, sequenceid=684, compaction requested=true 2023-03-19 09:55:00,172 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:00,172 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:00,179 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:00,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9f5006[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:00,181 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:00,181 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,182 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:00,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,187 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:00,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:00,189 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:00,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:00,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:00,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:00,189 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:00,191 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,191 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,191 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,191 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,193 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:00,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,195 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,195 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:00,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:00,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:55:00,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:00,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:00,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:00,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:00,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:00,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:00,295 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:00,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:00,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:00,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:00,299 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:00,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:00,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:00,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,301 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:00,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:00,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:00,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,303 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:00,303 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:00,303 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:00,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:00,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:00,303 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:00,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,305 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:00,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41df3a67[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:00,307 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:00,307 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:00,309 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,309 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,310 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:00,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:00,313 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:00,313 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:00,313 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:00,313 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:00,313 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:00,314 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:00,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:00,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:00,319 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,320 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:00,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,320 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:00,320 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,320 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,320 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219760332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219760358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219760359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219760361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219760366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,420 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:00,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:00,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:00,423 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:00,423 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,423 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:00,423 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:00,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,424 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:00,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:00,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:00,427 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,427 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:00,427 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:00,427 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:00,429 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:00,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2620e7d3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:00,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:00,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:00,434 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:00,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:00,436 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:00,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:00,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:00,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:00,437 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:00,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,439 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219760437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,440 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,440 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:00,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:00,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:00,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,443 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:00,443 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:00,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,543 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:00,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:00,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:00,546 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:00,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:00,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:00,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,548 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:00,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:00,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:00,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:00,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:00,550 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:00,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:00,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:00,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77280c27[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:00,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:00,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:00,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:00,557 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:00,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:00,560 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:00,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:00,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:00,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:00,560 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:00,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,567 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,570 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:00,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,570 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,570 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:00,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:00,570 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:00,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219760640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,670 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:00,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:00,674 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:00,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:00,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:00,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,679 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:00,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:00,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:00,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:00,683 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,683 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:00,683 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:00,684 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:00,685 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/8d386767b4674aafbbc202c5fdd7e915 2023-03-19 09:55:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:00,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46067da4[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:00,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:00,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,691 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:00,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:00,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:00,695 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:00,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:00,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:00,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:00,695 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:00,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:00,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:00,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,698 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,699 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:00,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:00,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:00,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:00,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:00,702 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:00,702 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:00,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/7077ac40fa0740bab1fb1776e4480ad5 2023-03-19 09:55:00,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/fe303c5cac3440a4b7ec632e6235e595 2023-03-19 09:55:00,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/8d386767b4674aafbbc202c5fdd7e915 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8d386767b4674aafbbc202c5fdd7e915 2023-03-19 09:55:00,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8d386767b4674aafbbc202c5fdd7e915, entries=200, sequenceid=701, filesize=14.3 K 2023-03-19 09:55:00,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/7077ac40fa0740bab1fb1776e4480ad5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/7077ac40fa0740bab1fb1776e4480ad5 2023-03-19 09:55:00,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/7077ac40fa0740bab1fb1776e4480ad5, entries=150, sequenceid=701, filesize=11.9 K 2023-03-19 09:55:00,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/fe303c5cac3440a4b7ec632e6235e595 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe303c5cac3440a4b7ec632e6235e595 2023-03-19 09:55:00,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe303c5cac3440a4b7ec632e6235e595, entries=150, sequenceid=701, filesize=11.9 K 2023-03-19 09:55:00,778 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8dad03d10fa8d87944de8392ea9480f0 in 500ms, sequenceid=701, compaction requested=true 2023-03-19 09:55:00,778 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:00,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:00,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:00,778 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:00,778 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:00,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:00,780 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64154 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:00,780 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:00,780 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:55:00,780 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:55:00,780 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,780 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,780 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8b9a6fd8c5d24108aea96098f105dbba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1479a193bdd147f5bc446b2e683e1de5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/babd198595a44f53b42a4c2ee94acc70, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/7077ac40fa0740bab1fb1776e4480ad5] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=60.3 K 2023-03-19 09:55:00,780 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/94edf94645b34afa9ad95a1bf748a9a2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/d225887b4d9e45dc956959d255032554, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/516c084a6cbe4a0f8a4c4c424a2f9800, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/589d53e2fc81420a81836eacfb5b8a53, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8d386767b4674aafbbc202c5fdd7e915] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=62.7 K 2023-03-19 09:55:00,781 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8b9a6fd8c5d24108aea96098f105dbba, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679219697281 2023-03-19 09:55:00,781 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 94edf94645b34afa9ad95a1bf748a9a2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679219697281 2023-03-19 09:55:00,781 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1479a193bdd147f5bc446b2e683e1de5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1679219697601 2023-03-19 09:55:00,781 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d225887b4d9e45dc956959d255032554, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1679219697601 2023-03-19 09:55:00,782 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting babd198595a44f53b42a4c2ee94acc70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1679219698494 2023-03-19 09:55:00,782 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 516c084a6cbe4a0f8a4c4c424a2f9800, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1679219698494 2023-03-19 09:55:00,785 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2ebb8bf5b1024eb1bc7edc2ad5685ce5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1679219699141 2023-03-19 09:55:00,785 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 589d53e2fc81420a81836eacfb5b8a53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1679219699141 2023-03-19 09:55:00,785 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 7077ac40fa0740bab1fb1776e4480ad5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679219699734 2023-03-19 09:55:00,786 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8d386767b4674aafbbc202c5fdd7e915, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679219699734 2023-03-19 09:55:00,801 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#279 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:00,804 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#280 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:00,806 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:00,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:00,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:00,816 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,819 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:00,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,820 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:00,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:00,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:00,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:00,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:00,832 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:00,833 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:00,833 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:00,833 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:55:00,833 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:00,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:00,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:00,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:00,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:00,834 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:00,840 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/b66a060c5acd4ebd829340feebbb6e03 2023-03-19 09:55:00,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:00,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:00,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219760879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219760879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219760880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219760881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219760945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219760983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219760983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219760983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:00,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219760985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219761186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219761186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219761187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219761188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,235 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/546cfb9ff841419a898970fff20bd393 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/546cfb9ff841419a898970fff20bd393 2023-03-19 09:55:01,237 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/f9c017585e094344a40bee2acc5f9a7c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f9c017585e094344a40bee2acc5f9a7c 2023-03-19 09:55:01,239 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 546cfb9ff841419a898970fff20bd393(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:01,239 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:01,239 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=11, startTime=1679219700778; duration=0sec 2023-03-19 09:55:01,240 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:01,240 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:01,241 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:01,241 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:55:01,241 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:01,242 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/9ad49f3562f14d3680575ef590d7f7b1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6c55f107d33949f38e49f511c79036c4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/66559e4f72e54e00ae8ed47cdf5b5769, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c6541b8544ff4b42970aca65a15c50c7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe303c5cac3440a4b7ec632e6235e595] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=60.2 K 2023-03-19 09:55:01,242 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9ad49f3562f14d3680575ef590d7f7b1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679219697281 2023-03-19 09:55:01,242 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into f9c017585e094344a40bee2acc5f9a7c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:01,242 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:01,242 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=11, startTime=1679219700778; duration=0sec 2023-03-19 09:55:01,242 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:01,243 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6c55f107d33949f38e49f511c79036c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1679219697601 2023-03-19 09:55:01,243 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 66559e4f72e54e00ae8ed47cdf5b5769, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1679219698494 2023-03-19 09:55:01,243 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c6541b8544ff4b42970aca65a15c50c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1679219699141 2023-03-19 09:55:01,244 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fe303c5cac3440a4b7ec632e6235e595, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679219699734 2023-03-19 09:55:01,250 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/85b2561cd60d4e8b9825f449c7609bdb 2023-03-19 09:55:01,252 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#283 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:01,259 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/b9fb033c87444e0a83c3079176540ea8 2023-03-19 09:55:01,263 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/b66a060c5acd4ebd829340feebbb6e03 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b66a060c5acd4ebd829340feebbb6e03 2023-03-19 09:55:01,266 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b66a060c5acd4ebd829340feebbb6e03, entries=150, sequenceid=716, filesize=11.9 K 2023-03-19 09:55:01,267 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/85b2561cd60d4e8b9825f449c7609bdb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/85b2561cd60d4e8b9825f449c7609bdb 2023-03-19 09:55:01,271 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/85b2561cd60d4e8b9825f449c7609bdb, entries=150, sequenceid=716, filesize=11.9 K 2023-03-19 09:55:01,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/b9fb033c87444e0a83c3079176540ea8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b9fb033c87444e0a83c3079176540ea8 2023-03-19 09:55:01,274 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b9fb033c87444e0a83c3079176540ea8, entries=150, sequenceid=716, filesize=11.9 K 2023-03-19 09:55:01,275 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8dad03d10fa8d87944de8392ea9480f0 in 442ms, sequenceid=716, compaction requested=false 2023-03-19 09:55:01,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:01,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:01,278 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:01,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:01,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:01,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:01,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:01,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e327b01[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:01,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:01,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:01,283 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,283 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:01,284 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:01,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:01,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:01,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:01,287 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:01,287 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:01,287 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:01,287 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:01,287 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:01,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:01,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:01,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:01,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:01,293 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:01,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:01,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:01,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:01,293 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:01,293 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:01,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:01,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:01,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:01,294 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:01,294 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:01,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:01,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:01,394 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:01,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:01,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:01,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:01,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:01,398 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:01,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:01,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:01,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:01,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:01,399 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:01,399 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:01,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:01,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:01,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:01,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:01,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:01,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:01,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:01,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:01,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:01,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:01,402 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:01,402 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:01,403 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:55:01,403 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:01,403 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:01,403 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:01,403 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:01,403 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:01,403 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:01,422 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/653a2584f2e541528070559ee6f6f3fe 2023-03-19 09:55:01,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:01,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:01,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219761490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219761492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219761492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219761492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219761492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219761596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,661 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/300551f752b34f92b055f8a406e79291 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/300551f752b34f92b055f8a406e79291 2023-03-19 09:55:01,665 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into 300551f752b34f92b055f8a406e79291(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:01,666 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:01,666 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=11, startTime=1679219700778; duration=0sec 2023-03-19 09:55:01,666 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:01,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219761802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,850 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/b994f2f3ec304c5880916ffe621de725 2023-03-19 09:55:01,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219761994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219761994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219761998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:01,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219761999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:02,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219762105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,262 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c15fcdade2a84534974e06c2ba007ce1 2023-03-19 09:55:02,266 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/653a2584f2e541528070559ee6f6f3fe as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/653a2584f2e541528070559ee6f6f3fe 2023-03-19 09:55:02,269 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/653a2584f2e541528070559ee6f6f3fe, entries=150, sequenceid=735, filesize=11.9 K 2023-03-19 09:55:02,270 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/b994f2f3ec304c5880916ffe621de725 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b994f2f3ec304c5880916ffe621de725 2023-03-19 09:55:02,274 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b994f2f3ec304c5880916ffe621de725, entries=150, sequenceid=735, filesize=11.9 K 2023-03-19 09:55:02,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c15fcdade2a84534974e06c2ba007ce1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c15fcdade2a84534974e06c2ba007ce1 2023-03-19 09:55:02,281 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c15fcdade2a84534974e06c2ba007ce1, entries=150, sequenceid=735, filesize=11.9 K 2023-03-19 09:55:02,282 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8dad03d10fa8d87944de8392ea9480f0 in 880ms, sequenceid=735, compaction requested=true 2023-03-19 09:55:02,282 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:02,282 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:02,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:02,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0d156b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:02,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:02,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:02,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:02,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:02,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:02,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:02,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:02,298 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:02,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:02,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:02,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:02,298 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:02,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,300 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,300 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:02,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:02,312 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:02,312 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,412 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:02,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:02,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:02,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:02,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:02,428 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:02,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:02,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:02,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,430 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:02,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:02,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:02,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:02,437 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:02,445 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/7a370eb1b0094501aa55bf2f686b8c84 2023-03-19 09:55:02,457 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/a0c6302d14f34d37a1c190e5b52b2b25 2023-03-19 09:55:02,470 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c93eca3e8f494a5b8d99d1da9dc7271e 2023-03-19 09:55:02,474 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/7a370eb1b0094501aa55bf2f686b8c84 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7a370eb1b0094501aa55bf2f686b8c84 2023-03-19 09:55:02,479 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7a370eb1b0094501aa55bf2f686b8c84, entries=150, sequenceid=751, filesize=11.9 K 2023-03-19 09:55:02,479 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-19 09:55:02,480 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/a0c6302d14f34d37a1c190e5b52b2b25 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a0c6302d14f34d37a1c190e5b52b2b25 2023-03-19 09:55:02,496 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a0c6302d14f34d37a1c190e5b52b2b25, entries=150, sequenceid=751, filesize=11.9 K 2023-03-19 09:55:02,497 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/c93eca3e8f494a5b8d99d1da9dc7271e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c93eca3e8f494a5b8d99d1da9dc7271e 2023-03-19 09:55:02,522 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c93eca3e8f494a5b8d99d1da9dc7271e, entries=150, sequenceid=751, filesize=11.9 K 2023-03-19 09:55:02,522 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 8dad03d10fa8d87944de8392ea9480f0 in 85ms, sequenceid=751, compaction requested=true 2023-03-19 09:55:02,523 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:02,523 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,524 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:02,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68736399[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:02,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:02,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:02,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:02,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:02,532 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:02,532 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:02,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:02,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:02,533 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:02,534 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,534 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,534 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,535 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,536 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:02,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:02,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,539 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:02,539 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:02,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:02,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:02,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:55:02,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:02,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:02,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:02,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:02,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:02,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:02,639 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:02,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:02,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:02,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:02,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:02,648 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:02,648 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,648 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:02,648 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:02,648 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,650 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:02,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:02,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:02,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:02,652 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,653 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:02,653 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:02,653 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:02,655 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,655 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dedd86e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:02,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:02,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,659 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:02,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:02,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:02,664 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:02,664 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:02,664 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:02,664 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:02,664 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:02,667 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,667 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,667 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,670 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:02,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:02,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:02,674 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:02,674 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:02,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219702684, max timestamp=1679219702702Num uniques 859; 2023-03-19 09:55:02,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219702684, max timestamp=1679219702702Num uniques 859; 2023-03-19 09:55:02,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:02,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:02,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219702684, max timestamp=1679219702702Num uniques 859; 2023-03-19 09:55:02,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:02,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219762767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,774 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:02,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:02,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:02,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:02,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:02,778 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:02,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:02,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:02,779 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,779 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,780 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,780 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:02,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:02,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:02,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:02,781 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:02,781 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:02,793 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,794 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:02,794 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:02,794 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,796 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:02,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ea42a1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:02,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:02,798 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,799 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:02,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,799 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,802 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:02,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:02,806 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:02,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:02,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:02,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:02,806 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:02,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,808 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,812 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,822 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,822 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:02,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:02,823 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,824 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:02,824 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:02,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:02,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:02,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219762874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,924 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:02,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:02,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:02,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:02,927 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:02,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:02,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:02,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,931 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,931 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:02,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:02,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:02,944 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,944 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:02,944 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:02,944 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:02,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,947 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:02,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477bb79a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:02,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:02,950 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:02,951 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:02,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:02,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:02,954 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:02,957 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:02,957 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:02,957 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:02,957 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:02,957 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:02,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:02,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,969 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:02,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:02,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:02,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:02,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:02,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:02,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:02,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:02,980 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:02,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:02,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:02,980 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:02,980 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:02,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:03,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:03,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219763002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:03,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219763003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:03,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219763004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:03,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219763006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,035 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/7b29dd539fd8476e9b5ae7794951f26e 2023-03-19 09:55:03,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/c83f711dbae840c7869aeda0a524c925 2023-03-19 09:55:03,080 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:03,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:03,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:03,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:03,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:03,084 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:03,084 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,084 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:03,084 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:03,084 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,084 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,085 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,086 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,087 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,087 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:03,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:03,088 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:03,089 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,089 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,090 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,090 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:03,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e76b47[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:03,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,095 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:03,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:03,096 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:03,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:03,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:03,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:03,098 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:03,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219763096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:03,101 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:03,101 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:03,101 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:03,101 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:03,101 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:03,103 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,103 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,103 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,104 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,105 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,107 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,108 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:03,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:03,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,108 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:03,108 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:03,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:03,208 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:03,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:03,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:03,212 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:03,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:03,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:03,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,214 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:03,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:03,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:03,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:03,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:03,217 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:03,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,219 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47baeb1d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:03,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:03,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:03,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:03,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:03,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:03,225 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:03,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:03,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:03,227 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:03,227 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:03,227 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:03,227 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:03,227 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:03,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,231 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,234 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:03,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:03,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,235 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:03,235 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:03,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,335 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:03,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:03,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:03,338 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:03,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:03,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:03,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,340 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:03,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:03,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:03,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:03,342 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,342 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:03,342 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:03,342 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:03,351 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e75601[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:03,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:03,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:03,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:03,358 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:03,358 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:03,358 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:03,358 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:03,358 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:03,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:03,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:03,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,366 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:03,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,366 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:03,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:03,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219763401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,466 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:03,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:03,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:03,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:03,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:03,470 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:03,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:03,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:03,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,472 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,473 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,473 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:03,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:03,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:03,474 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:03,474 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:03,474 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:03,474 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:03,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:03,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:03,477 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,477 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/7643c3f7feaf43949237ef3c5dec48d5 2023-03-19 09:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3aea40[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:03,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:03,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,481 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:03,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:03,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/7b29dd539fd8476e9b5ae7794951f26e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7b29dd539fd8476e9b5ae7794951f26e 2023-03-19 09:55:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:03,484 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:03,488 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:03,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:03,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:03,488 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:03,488 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:03,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7b29dd539fd8476e9b5ae7794951f26e, entries=200, sequenceid=761, filesize=14.3 K 2023-03-19 09:55:03,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/c83f711dbae840c7869aeda0a524c925 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c83f711dbae840c7869aeda0a524c925 2023-03-19 09:55:03,490 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,490 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,490 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:03,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:03,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,493 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:03,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:03,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:03,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:03,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:03,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,497 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:03,497 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:03,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:03,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:03,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c83f711dbae840c7869aeda0a524c925, entries=150, sequenceid=761, filesize=11.9 K 2023-03-19 09:55:03,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/7643c3f7feaf43949237ef3c5dec48d5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/7643c3f7feaf43949237ef3c5dec48d5 2023-03-19 09:55:03,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/7643c3f7feaf43949237ef3c5dec48d5, entries=150, sequenceid=761, filesize=11.9 K 2023-03-19 09:55:03,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8dad03d10fa8d87944de8392ea9480f0 in 887ms, sequenceid=761, compaction requested=true 2023-03-19 09:55:03,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:03,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:03,510 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:03,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:03,511 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-19 09:55:03,511 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:03,512 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64324 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:03,513 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:55:03,513 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,513 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f9c017585e094344a40bee2acc5f9a7c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b66a060c5acd4ebd829340feebbb6e03, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/653a2584f2e541528070559ee6f6f3fe, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7a370eb1b0094501aa55bf2f686b8c84, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7b29dd539fd8476e9b5ae7794951f26e] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=62.8 K 2023-03-19 09:55:03,513 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:03,513 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:55:03,513 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting f9c017585e094344a40bee2acc5f9a7c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679219699734 2023-03-19 09:55:03,513 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,514 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/546cfb9ff841419a898970fff20bd393, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/85b2561cd60d4e8b9825f449c7609bdb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b994f2f3ec304c5880916ffe621de725, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a0c6302d14f34d37a1c190e5b52b2b25, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c83f711dbae840c7869aeda0a524c925] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=60.4 K 2023-03-19 09:55:03,514 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b66a060c5acd4ebd829340feebbb6e03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1679219700307 2023-03-19 09:55:03,514 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 546cfb9ff841419a898970fff20bd393, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679219699734 2023-03-19 09:55:03,514 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 653a2584f2e541528070559ee6f6f3fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1679219700878 2023-03-19 09:55:03,514 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 85b2561cd60d4e8b9825f449c7609bdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1679219700307 2023-03-19 09:55:03,515 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7a370eb1b0094501aa55bf2f686b8c84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679219701466 2023-03-19 09:55:03,515 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b994f2f3ec304c5880916ffe621de725, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1679219700878 2023-03-19 09:55:03,515 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7b29dd539fd8476e9b5ae7794951f26e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=761, earliestPutTs=1679219702608 2023-03-19 09:55:03,515 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a0c6302d14f34d37a1c190e5b52b2b25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679219701466 2023-03-19 09:55:03,516 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c83f711dbae840c7869aeda0a524c925, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=761, earliestPutTs=1679219702608 2023-03-19 09:55:03,530 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#294 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:03,530 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#295 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:03,543 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/410bc6e72dbe4c7fbc6d409ed24b9390 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/410bc6e72dbe4c7fbc6d409ed24b9390 2023-03-19 09:55:03,546 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 410bc6e72dbe4c7fbc6d409ed24b9390(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:03,546 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:03,546 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=11, startTime=1679219703510; duration=0sec 2023-03-19 09:55:03,547 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:03,547 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:03,548 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:03,548 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:55:03,548 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,548 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/300551f752b34f92b055f8a406e79291, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b9fb033c87444e0a83c3079176540ea8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c15fcdade2a84534974e06c2ba007ce1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c93eca3e8f494a5b8d99d1da9dc7271e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/7643c3f7feaf43949237ef3c5dec48d5] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=60.4 K 2023-03-19 09:55:03,549 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 300551f752b34f92b055f8a406e79291, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679219699734 2023-03-19 09:55:03,549 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b9fb033c87444e0a83c3079176540ea8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1679219700307 2023-03-19 09:55:03,549 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c15fcdade2a84534974e06c2ba007ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1679219700878 2023-03-19 09:55:03,550 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c93eca3e8f494a5b8d99d1da9dc7271e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1679219701466 2023-03-19 09:55:03,550 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7643c3f7feaf43949237ef3c5dec48d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=761, earliestPutTs=1679219702608 2023-03-19 09:55:03,559 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#296 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:03,597 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:03,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:03,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:03,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:03,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:03,600 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:03,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:03,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:03,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,602 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,602 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:03,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:03,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:03,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:03,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:03,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:03,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:03,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:03,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:03,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:03,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:03,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:03,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:03,604 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:55:03,604 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:03,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:03,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:03,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:03,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219763924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:03,943 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/4fc0c79282aa4fd4b151ac3fadc8b085 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/4fc0c79282aa4fd4b151ac3fadc8b085 2023-03-19 09:55:03,947 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 4fc0c79282aa4fd4b151ac3fadc8b085(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:03,948 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:03,948 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=11, startTime=1679219703510; duration=0sec 2023-03-19 09:55:03,948 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:03,967 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/1183eb1522d2487b8991e94ff2ec8749 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/1183eb1522d2487b8991e94ff2ec8749 2023-03-19 09:55:03,971 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into 1183eb1522d2487b8991e94ff2ec8749(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:03,971 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:03,971 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=11, startTime=1679219703511; duration=0sec 2023-03-19 09:55:03,971 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:04,013 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/036c86a08e33434489312aad5b325940 2023-03-19 09:55:04,023 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/c04b3fdc78c0479e9d926c56592cf00d 2023-03-19 09:55:04,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:04,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219764028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,037 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/3cfc42e734524ad2892ae622806e5e63 2023-03-19 09:55:04,040 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/036c86a08e33434489312aad5b325940 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/036c86a08e33434489312aad5b325940 2023-03-19 09:55:04,043 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/036c86a08e33434489312aad5b325940, entries=150, sequenceid=799, filesize=11.9 K 2023-03-19 09:55:04,044 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/c04b3fdc78c0479e9d926c56592cf00d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c04b3fdc78c0479e9d926c56592cf00d 2023-03-19 09:55:04,047 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c04b3fdc78c0479e9d926c56592cf00d, entries=150, sequenceid=799, filesize=11.9 K 2023-03-19 09:55:04,048 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/3cfc42e734524ad2892ae622806e5e63 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3cfc42e734524ad2892ae622806e5e63 2023-03-19 09:55:04,053 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3cfc42e734524ad2892ae622806e5e63, entries=150, sequenceid=799, filesize=11.9 K 2023-03-19 09:55:04,054 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8dad03d10fa8d87944de8392ea9480f0 in 450ms, sequenceid=799, compaction requested=false 2023-03-19 09:55:04,054 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:04,054 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:04,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,059 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,059 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:04,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70881bb3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:04,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:04,061 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,062 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:04,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:04,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:04,062 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,062 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:04,064 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:04,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:04,067 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:04,067 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:04,067 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:04,067 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:04,067 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:04,069 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,069 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,069 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,070 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,072 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:04,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:04,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:04,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:04,075 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:04,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:04,075 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:04,075 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:04,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,175 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:04,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:04,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:04,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:04,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:04,178 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:04,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:04,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:04,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:04,179 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,179 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:04,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:04,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:04,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:04,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:04,182 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:04,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:04,193 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=811 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/ae66dd0dbfe74afab45c799839dff645 2023-03-19 09:55:04,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:04,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:04,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219704239, max timestamp=1679219704268Num uniques 859; 2023-03-19 09:55:04,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219704239, max timestamp=1679219704268Num uniques 859; 2023-03-19 09:55:04,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:04,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219704239, max timestamp=1679219704268Num uniques 859; 2023-03-19 09:55:04,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:04,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:04,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:04,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219764314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:04,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219764417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,609 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=811 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2746306c3a844b6081259ea0256ec001 2023-03-19 09:55:04,619 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=811 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/57776c9bd38a4236842a110600aa59a6 2023-03-19 09:55:04,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:04,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219764619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,623 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/ae66dd0dbfe74afab45c799839dff645 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/ae66dd0dbfe74afab45c799839dff645 2023-03-19 09:55:04,626 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/ae66dd0dbfe74afab45c799839dff645, entries=150, sequenceid=811, filesize=11.9 K 2023-03-19 09:55:04,627 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2746306c3a844b6081259ea0256ec001 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2746306c3a844b6081259ea0256ec001 2023-03-19 09:55:04,630 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2746306c3a844b6081259ea0256ec001, entries=150, sequenceid=811, filesize=11.9 K 2023-03-19 09:55:04,630 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/57776c9bd38a4236842a110600aa59a6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/57776c9bd38a4236842a110600aa59a6 2023-03-19 09:55:04,634 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/57776c9bd38a4236842a110600aa59a6, entries=150, sequenceid=811, filesize=11.9 K 2023-03-19 09:55:04,635 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8dad03d10fa8d87944de8392ea9480f0 in 453ms, sequenceid=811, compaction requested=true 2023-03-19 09:55:04,635 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:04,635 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:04,635 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:04,637 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,637 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,638 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,638 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:04,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60fae4bc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:04,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:04,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,643 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:04,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:04,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:04,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:04,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:04,645 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:04,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:04,647 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:04,647 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:04,647 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:04,647 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:04,647 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:04,649 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,649 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,649 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,653 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:04,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:04,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:04,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:04,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:04,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,654 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:04,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:04,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,654 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,654 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,654 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,654 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:04,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,754 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:04,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:04,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:04,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:04,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:04,757 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:04,757 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:04,757 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:04,757 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:04,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,760 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:04,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:04,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:04,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:04,761 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:04,761 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:04,762 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:04,762 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:04,762 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:55:04,762 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:04,762 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:04,762 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:04,762 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:04,763 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:04,763 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:04,776 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=848 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/cdfe179b889b43b08f7fd5114bbddcf3 2023-03-19 09:55:04,786 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=848 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2c3a9b063fe24bc1b411d13d224be994 2023-03-19 09:55:04,804 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=848 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/08f2cdc744e14b14813586947e1ecc63 2023-03-19 09:55:04,809 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/cdfe179b889b43b08f7fd5114bbddcf3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/cdfe179b889b43b08f7fd5114bbddcf3 2023-03-19 09:55:04,813 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/cdfe179b889b43b08f7fd5114bbddcf3, entries=150, sequenceid=848, filesize=11.9 K 2023-03-19 09:55:04,814 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2c3a9b063fe24bc1b411d13d224be994 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2c3a9b063fe24bc1b411d13d224be994 2023-03-19 09:55:04,817 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2c3a9b063fe24bc1b411d13d224be994, entries=150, sequenceid=848, filesize=11.9 K 2023-03-19 09:55:04,818 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/08f2cdc744e14b14813586947e1ecc63 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/08f2cdc744e14b14813586947e1ecc63 2023-03-19 09:55:04,822 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/08f2cdc744e14b14813586947e1ecc63, entries=150, sequenceid=848, filesize=11.9 K 2023-03-19 09:55:04,824 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 8dad03d10fa8d87944de8392ea9480f0 in 62ms, sequenceid=848, compaction requested=true 2023-03-19 09:55:04,824 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:04,824 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:04,824 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:04,824 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:04,824 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,824 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:04,824 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:04,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,826 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:04,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f72e1fc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:04,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:04,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,832 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:04,832 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,832 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:04,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:04,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:04,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:04,837 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:04,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:04,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:04,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:04,837 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:04,840 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,840 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,840 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,841 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,842 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:04,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:04,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,845 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:04,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,845 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:04,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:04,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:55:04,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:04,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:04,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:04,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:04,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:04,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:04,946 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:04,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:04,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:04,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:04,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:04,949 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:04,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:04,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:04,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:04,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,949 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:04,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:04,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:04,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:04,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:04,952 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:04,952 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:04,952 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:04,952 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:04,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:04,954 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:04,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173274b8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:04,956 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:04,956 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:04,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:04,960 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:04,960 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:04,962 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:04,965 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:04,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:04,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:04,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:04,965 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:04,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:04,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:04,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,969 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:04,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:04,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:04,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:04,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:04,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:04,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:04,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:04,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:04,972 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:04,972 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:04,972 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:04,972 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:04,972 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:04,972 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219704987, max timestamp=1679219705001Num uniques 859; 2023-03-19 09:55:05,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219704987, max timestamp=1679219705001Num uniques 859; 2023-03-19 09:55:05,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:05,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:05,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219704987, max timestamp=1679219705001Num uniques 859; 2023-03-19 09:55:05,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:05,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219765026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219765027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219765027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219765031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219765031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,072 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:05,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:05,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:05,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:05,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:05,075 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:05,075 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:05,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:05,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,079 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:05,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:05,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:05,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:05,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:05,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:05,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:05,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:05,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:05,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:05,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,086 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:05,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d7b06e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:05,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:05,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,090 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,090 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:05,090 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:05,090 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:05,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:05,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:05,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:05,091 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:05,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:05,095 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:05,095 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:05,095 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:05,095 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:05,095 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:05,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,105 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:05,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:05,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,111 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,111 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:05,111 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:05,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219765133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219765133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219765133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219765136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219765136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,211 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:05,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:05,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:05,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:05,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:05,214 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:05,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:05,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:05,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:05,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:05,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:05,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:05,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:05,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:05,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:05,218 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:05,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:05,220 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,220 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:05,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:05,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10bf6f2c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:05,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:05,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:05,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:05,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:05,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:05,226 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:05,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:05,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:05,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:05,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:05,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:05,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:05,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,230 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:05,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:05,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,235 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,235 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,335 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:05,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:05,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:05,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219765335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219765337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219765337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:05,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:05,340 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:05,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:05,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:05,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219765339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219765340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:05,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:05,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:05,348 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=858 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/0dcf61aaab3c4010a6519adffa291a1c 2023-03-19 09:55:05,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:05,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:05,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:05,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:05,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:05,354 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,354 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:05,354 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:05,354 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:05,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ea01c6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:05,358 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:05,358 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,359 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,359 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:05,359 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:05,359 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:05,359 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,359 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:05,361 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=858 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/ab42d2e31e184348ba2d26b68fe3ed0c 2023-03-19 09:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:05,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:05,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:05,365 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:05,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:05,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:05,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:05,365 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:05,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,368 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:05,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,372 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,372 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:05,372 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:05,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,472 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:05,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:05,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:05,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:05,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:05,475 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:05,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:05,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:05,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,477 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,477 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:05,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:05,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:05,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:05,479 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,479 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:05,479 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:05,479 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:05,479 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:05,481 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,481 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,481 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,481 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:05,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7935d7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:05,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:05,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,485 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,485 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:05,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:05,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,487 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:05,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:05,489 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:05,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:05,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:05,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:05,489 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:05,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:05,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:05,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,499 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:05,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,500 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:05,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,600 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:05,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:05,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:05,603 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:05,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:05,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:05,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,603 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,604 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,604 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:05,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:05,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:05,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:05,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:05,606 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:05,606 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,606 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:05,606 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:05,606 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:05,608 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b7b6f6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:05,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:05,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:05,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:05,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:05,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:05,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:05,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:05,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:05,618 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:05,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:05,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:05,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:05,618 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:05,620 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,620 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,620 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,620 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,625 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:05,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:05,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,625 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,625 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:05,625 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:05,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35896 deadline: 1679219765639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35918 deadline: 1679219765640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35938 deadline: 1679219765640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35930 deadline: 1679219765644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35946 deadline: 1679219765644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,726 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:05,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:05,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:05,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:05,729 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:05,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:05,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:05,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,732 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:05,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:05,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:05,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:05,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:05,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:05,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:05,733 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:05,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. started... 2023-03-19 09:55:05,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. as already flushing 2023-03-19 09:55:05,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:05,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:05,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1a7b03[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:05,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:05,755 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,757 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:05,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:05,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:05,757 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,757 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:05,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,759 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:05,759 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,759 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:05,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:05,761 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:05,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:05,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:05,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:05,761 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:05,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:05,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,771 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:05,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:05,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:05,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:05,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:05,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:05,777 DEBUG [Thread-1604] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:05,777 DEBUG [Thread-1604] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:05,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:05,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=858 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/3c23a7e57f47476889e644cd1103dd79 2023-03-19 09:55:05,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/0dcf61aaab3c4010a6519adffa291a1c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/0dcf61aaab3c4010a6519adffa291a1c 2023-03-19 09:55:05,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/0dcf61aaab3c4010a6519adffa291a1c, entries=100, sequenceid=858, filesize=9.5 K 2023-03-19 09:55:05,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/ab42d2e31e184348ba2d26b68fe3ed0c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ab42d2e31e184348ba2d26b68fe3ed0c 2023-03-19 09:55:05,794 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ab42d2e31e184348ba2d26b68fe3ed0c, entries=100, sequenceid=858, filesize=9.5 K 2023-03-19 09:55:05,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/3c23a7e57f47476889e644cd1103dd79 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3c23a7e57f47476889e644cd1103dd79 2023-03-19 09:55:05,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3c23a7e57f47476889e644cd1103dd79, entries=100, sequenceid=858, filesize=9.5 K 2023-03-19 09:55:05,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8dad03d10fa8d87944de8392ea9480f0 in 860ms, sequenceid=858, compaction requested=true 2023-03-19 09:55:05,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:05,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:05,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:05,798 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:05,798 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:05,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:05,800 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59610 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:05,800 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/B is initiating minor compaction (all files) 2023-03-19 09:55:05,800 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59610 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:05,800 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/B in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,800 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/A is initiating minor compaction (all files) 2023-03-19 09:55:05,800 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/A in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,800 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/4fc0c79282aa4fd4b151ac3fadc8b085, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c04b3fdc78c0479e9d926c56592cf00d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2746306c3a844b6081259ea0256ec001, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2c3a9b063fe24bc1b411d13d224be994, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ab42d2e31e184348ba2d26b68fe3ed0c] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=58.2 K 2023-03-19 09:55:05,800 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/410bc6e72dbe4c7fbc6d409ed24b9390, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/036c86a08e33434489312aad5b325940, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/ae66dd0dbfe74afab45c799839dff645, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/cdfe179b889b43b08f7fd5114bbddcf3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/0dcf61aaab3c4010a6519adffa291a1c] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=58.2 K 2023-03-19 09:55:05,800 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4fc0c79282aa4fd4b151ac3fadc8b085, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=761, earliestPutTs=1679219702608 2023-03-19 09:55:05,800 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 410bc6e72dbe4c7fbc6d409ed24b9390, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=761, earliestPutTs=1679219702608 2023-03-19 09:55:05,801 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c04b3fdc78c0479e9d926c56592cf00d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1679219702751 2023-03-19 09:55:05,801 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 036c86a08e33434489312aad5b325940, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1679219702751 2023-03-19 09:55:05,801 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2746306c3a844b6081259ea0256ec001, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=811, earliestPutTs=1679219703911 2023-03-19 09:55:05,801 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ae66dd0dbfe74afab45c799839dff645, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=811, earliestPutTs=1679219703911 2023-03-19 09:55:05,801 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2c3a9b063fe24bc1b411d13d224be994, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=848, earliestPutTs=1679219704307 2023-03-19 09:55:05,801 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting cdfe179b889b43b08f7fd5114bbddcf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=848, earliestPutTs=1679219704307 2023-03-19 09:55:05,802 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ab42d2e31e184348ba2d26b68fe3ed0c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=858, earliestPutTs=1679219704931 2023-03-19 09:55:05,802 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0dcf61aaab3c4010a6519adffa291a1c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=858, earliestPutTs=1679219704931 2023-03-19 09:55:05,810 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#A#compaction#310 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:05,811 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#B#compaction#309 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:05,819 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/9b8d84eb98fe4fc7a24edaefc6c2da6f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9b8d84eb98fe4fc7a24edaefc6c2da6f 2023-03-19 09:55:05,822 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/A of 8dad03d10fa8d87944de8392ea9480f0 into 9b8d84eb98fe4fc7a24edaefc6c2da6f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:05,822 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:05,822 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/A, priority=11, startTime=1679219705798; duration=0sec 2023-03-19 09:55:05,822 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:05,822 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:05,823 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59576 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:05,824 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 8dad03d10fa8d87944de8392ea9480f0/C is initiating minor compaction (all files) 2023-03-19 09:55:05,824 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dad03d10fa8d87944de8392ea9480f0/C in TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:05,824 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/1183eb1522d2487b8991e94ff2ec8749, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3cfc42e734524ad2892ae622806e5e63, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/57776c9bd38a4236842a110600aa59a6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/08f2cdc744e14b14813586947e1ecc63, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3c23a7e57f47476889e644cd1103dd79] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp, totalSize=58.2 K 2023-03-19 09:55:05,824 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1183eb1522d2487b8991e94ff2ec8749, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=761, earliestPutTs=1679219702608 2023-03-19 09:55:05,824 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3cfc42e734524ad2892ae622806e5e63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1679219702751 2023-03-19 09:55:05,825 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 57776c9bd38a4236842a110600aa59a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=811, earliestPutTs=1679219703911 2023-03-19 09:55:05,825 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 08f2cdc744e14b14813586947e1ecc63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=848, earliestPutTs=1679219704307 2023-03-19 09:55:05,825 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3c23a7e57f47476889e644cd1103dd79, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=858, earliestPutTs=1679219704931 2023-03-19 09:55:05,833 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dad03d10fa8d87944de8392ea9480f0#C#compaction#311 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:05,843 DEBUG [Thread-1613] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:54245 2023-03-19 09:55:05,844 DEBUG [Thread-1613] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:05,844 DEBUG [Thread-1605] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:54245 2023-03-19 09:55:05,844 DEBUG [Thread-1605] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:05,845 DEBUG [Thread-1607] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:54245 2023-03-19 09:55:05,845 DEBUG [Thread-1607] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:05,846 DEBUG [Thread-1611] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:54245 2023-03-19 09:55:05,846 DEBUG [Thread-1611] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:05,847 DEBUG [Thread-1609] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:54245 2023-03-19 09:55:05,847 DEBUG [Thread-1609] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:05,878 DEBUG [Thread-1604] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:05,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:06,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:06,147 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=147.60 KB heapSize=425.53 KB 2023-03-19 09:55:06,147 DEBUG [Thread-1598] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:54245 2023-03-19 09:55:06,148 DEBUG [Thread-1598] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:06,148 DEBUG [Thread-1594] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:54245 2023-03-19 09:55:06,148 DEBUG [Thread-1594] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:06,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:06,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:55:06,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:06,148 DEBUG [Thread-1596] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:54245 2023-03-19 09:55:06,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:55:06,148 DEBUG [Thread-1596] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:06,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:06,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:55:06,151 DEBUG [Thread-1600] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:54245 2023-03-19 09:55:06,151 DEBUG [Thread-1600] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:06,152 DEBUG [Thread-1602] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:54245 2023-03-19 09:55:06,152 DEBUG [Thread-1602] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:06,152 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-19 09:55:06,152 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-03-19 09:55:06,152 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-03-19 09:55:06,152 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-03-19 09:55:06,152 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 241 2023-03-19 09:55:06,152 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-03-19 09:55:06,152 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 5151 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 5016 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 5042 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 5148 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 5038 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-19 09:55:06,153 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:54245 2023-03-19 09:55:06,153 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:06,153 INFO [Listener at localhost/45961] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-19 09:55:06,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-19 09:55:06,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:06,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-19 09:55:06,157 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219706157"}]},"ts":"1679219706157"} 2023-03-19 09:55:06,158 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-19 09:55:06,159 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-19 09:55:06,160 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dad03d10fa8d87944de8392ea9480f0, UNASSIGN}] 2023-03-19 09:55:06,161 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dad03d10fa8d87944de8392ea9480f0, UNASSIGN 2023-03-19 09:55:06,161 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=8dad03d10fa8d87944de8392ea9480f0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:06,161 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219706161"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219706161"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219706161"}]},"ts":"1679219706161"} 2023-03-19 09:55:06,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:55:06,219 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/399837b847884b6f9a1275400caa4a4e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/399837b847884b6f9a1275400caa4a4e 2023-03-19 09:55:06,222 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/B of 8dad03d10fa8d87944de8392ea9480f0 into 399837b847884b6f9a1275400caa4a4e(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:06,222 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:06,222 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/B, priority=11, startTime=1679219705798; duration=0sec 2023-03-19 09:55:06,222 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:06,240 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/01811017af4f45fdaa6e7010971186a0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/01811017af4f45fdaa6e7010971186a0 2023-03-19 09:55:06,243 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dad03d10fa8d87944de8392ea9480f0/C of 8dad03d10fa8d87944de8392ea9480f0 into 01811017af4f45fdaa6e7010971186a0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:06,243 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:06,243 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0., storeName=8dad03d10fa8d87944de8392ea9480f0/C, priority=11, startTime=1679219705798; duration=0sec 2023-03-19 09:55:06,243 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:06,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-19 09:55:06,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:06,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8dad03d10fa8d87944de8392ea9480f0, disabling compactions & flushes 2023-03-19 09:55:06,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:06,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-19 09:55:06,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/dabcb1c6e4354dcf8a32ce5de17b8899 2023-03-19 09:55:06,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-19 09:55:06,966 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2db901d57cc24005b354ba4bd9fc9e66 2023-03-19 09:55:07,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-19 09:55:07,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/84aba8cb46414ac1ab51e83d6aba7510 2023-03-19 09:55:07,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/dabcb1c6e4354dcf8a32ce5de17b8899 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dabcb1c6e4354dcf8a32ce5de17b8899 2023-03-19 09:55:07,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dabcb1c6e4354dcf8a32ce5de17b8899, entries=150, sequenceid=899, filesize=11.9 K 2023-03-19 09:55:07,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/2db901d57cc24005b354ba4bd9fc9e66 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2db901d57cc24005b354ba4bd9fc9e66 2023-03-19 09:55:07,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2db901d57cc24005b354ba4bd9fc9e66, entries=150, sequenceid=899, filesize=11.9 K 2023-03-19 09:55:07,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/84aba8cb46414ac1ab51e83d6aba7510 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/84aba8cb46414ac1ab51e83d6aba7510 2023-03-19 09:55:07,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/84aba8cb46414ac1ab51e83d6aba7510, entries=150, sequenceid=899, filesize=11.9 K 2023-03-19 09:55:07,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~445.41 KB/456096, currentSize=13.42 KB/13740 for 8dad03d10fa8d87944de8392ea9480f0 in 1241ms, sequenceid=899, compaction requested=false 2023-03-19 09:55:07,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:07,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. after waiting 0 ms 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:07,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8dad03d10fa8d87944de8392ea9480f0 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=A 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=B 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dad03d10fa8d87944de8392ea9480f0, store=C 2023-03-19 09:55:07,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:07,794 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/6e85db48590f4d3c916c44e199810a60 2023-03-19 09:55:08,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/fbf1f686517b4564b5caac6edef2d095 2023-03-19 09:55:08,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-19 09:55:08,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/f05a5ca514084ce1b26c36b8194d6624 2023-03-19 09:55:08,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/A/6e85db48590f4d3c916c44e199810a60 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/6e85db48590f4d3c916c44e199810a60 2023-03-19 09:55:08,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/6e85db48590f4d3c916c44e199810a60, entries=100, sequenceid=906, filesize=9.5 K 2023-03-19 09:55:08,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/B/fbf1f686517b4564b5caac6edef2d095 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/fbf1f686517b4564b5caac6edef2d095 2023-03-19 09:55:08,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/fbf1f686517b4564b5caac6edef2d095, entries=100, sequenceid=906, filesize=9.5 K 2023-03-19 09:55:08,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/.tmp/C/f05a5ca514084ce1b26c36b8194d6624 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f05a5ca514084ce1b26c36b8194d6624 2023-03-19 09:55:08,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f05a5ca514084ce1b26c36b8194d6624, entries=100, sequenceid=906, filesize=9.5 K 2023-03-19 09:55:08,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 8dad03d10fa8d87944de8392ea9480f0 in 1237ms, sequenceid=906, compaction requested=true 2023-03-19 09:55:08,646 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/3c17e5ea49d54856b35e18ac94e82007, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/024e7adbeafe4ae5a6f0e8c53aa4dd40, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d80f2c6f5a541958fe890f837afac69, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8848450d1d8342ddbd6aa5d90273a94b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/633ce4e581c74ad3b3c33f0a0f451057, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c823958f705f41be8975b28c2413bf46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/000bad7d9bdc4b9f97823f54e686d644, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b234541df3f04e2ba7fc2486f8a578b9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8a71b8debcd0427883a299240f466bf6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/75c78666f49c42f5bc87a01087aadf46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dd41ca664cb84bf794cdd3f306431487, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dc2660baa6e54533b904e6ece9f64a3e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7e74d6b6257e44029cf141633c34d1b1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/1c5f30a01b2e4f949d77c6957cb34e50, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/19cfd70fd0c745f4b3cfbc838a008f4a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d96839610e44f389a9cf942a3f8bdb3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5d3f043ed2b242bbbe2c9ec1adc09d9b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/95d060f48d7547ce8d9e6876416c55b8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5cbdfae384c14cdb8f3e7e38309c74d0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f602f3de205f408fb8b59b647b97a57f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b364d9ece9e34316998c17732d0f3b8d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/94edf94645b34afa9ad95a1bf748a9a2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c5d865bf5a83489a8cd85475fe4935e4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/d225887b4d9e45dc956959d255032554, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/516c084a6cbe4a0f8a4c4c424a2f9800, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/589d53e2fc81420a81836eacfb5b8a53, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8d386767b4674aafbbc202c5fdd7e915, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f9c017585e094344a40bee2acc5f9a7c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b66a060c5acd4ebd829340feebbb6e03, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/653a2584f2e541528070559ee6f6f3fe, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7a370eb1b0094501aa55bf2f686b8c84, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7b29dd539fd8476e9b5ae7794951f26e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/410bc6e72dbe4c7fbc6d409ed24b9390, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/036c86a08e33434489312aad5b325940, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/ae66dd0dbfe74afab45c799839dff645, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/cdfe179b889b43b08f7fd5114bbddcf3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/0dcf61aaab3c4010a6519adffa291a1c] to archive 2023-03-19 09:55:08,647 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:55:08,648 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/3c17e5ea49d54856b35e18ac94e82007 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/3c17e5ea49d54856b35e18ac94e82007 2023-03-19 09:55:08,649 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/024e7adbeafe4ae5a6f0e8c53aa4dd40 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/024e7adbeafe4ae5a6f0e8c53aa4dd40 2023-03-19 09:55:08,650 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d80f2c6f5a541958fe890f837afac69 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d80f2c6f5a541958fe890f837afac69 2023-03-19 09:55:08,651 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8848450d1d8342ddbd6aa5d90273a94b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8848450d1d8342ddbd6aa5d90273a94b 2023-03-19 09:55:08,652 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/633ce4e581c74ad3b3c33f0a0f451057 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/633ce4e581c74ad3b3c33f0a0f451057 2023-03-19 09:55:08,653 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c823958f705f41be8975b28c2413bf46 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c823958f705f41be8975b28c2413bf46 2023-03-19 09:55:08,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/000bad7d9bdc4b9f97823f54e686d644 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/000bad7d9bdc4b9f97823f54e686d644 2023-03-19 09:55:08,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b234541df3f04e2ba7fc2486f8a578b9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b234541df3f04e2ba7fc2486f8a578b9 2023-03-19 09:55:08,655 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8a71b8debcd0427883a299240f466bf6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8a71b8debcd0427883a299240f466bf6 2023-03-19 09:55:08,656 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/75c78666f49c42f5bc87a01087aadf46 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/75c78666f49c42f5bc87a01087aadf46 2023-03-19 09:55:08,657 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dd41ca664cb84bf794cdd3f306431487 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dd41ca664cb84bf794cdd3f306431487 2023-03-19 09:55:08,658 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dc2660baa6e54533b904e6ece9f64a3e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dc2660baa6e54533b904e6ece9f64a3e 2023-03-19 09:55:08,659 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7e74d6b6257e44029cf141633c34d1b1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7e74d6b6257e44029cf141633c34d1b1 2023-03-19 09:55:08,660 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/1c5f30a01b2e4f949d77c6957cb34e50 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/1c5f30a01b2e4f949d77c6957cb34e50 2023-03-19 09:55:08,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/18fe04bb8d1749b6ab7b2c4ddf63ffbd 2023-03-19 09:55:08,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/19cfd70fd0c745f4b3cfbc838a008f4a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/19cfd70fd0c745f4b3cfbc838a008f4a 2023-03-19 09:55:08,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d96839610e44f389a9cf942a3f8bdb3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9d96839610e44f389a9cf942a3f8bdb3 2023-03-19 09:55:08,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5d3f043ed2b242bbbe2c9ec1adc09d9b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5d3f043ed2b242bbbe2c9ec1adc09d9b 2023-03-19 09:55:08,664 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/95d060f48d7547ce8d9e6876416c55b8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/95d060f48d7547ce8d9e6876416c55b8 2023-03-19 09:55:08,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5cbdfae384c14cdb8f3e7e38309c74d0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/5cbdfae384c14cdb8f3e7e38309c74d0 2023-03-19 09:55:08,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f602f3de205f408fb8b59b647b97a57f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f602f3de205f408fb8b59b647b97a57f 2023-03-19 09:55:08,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b364d9ece9e34316998c17732d0f3b8d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b364d9ece9e34316998c17732d0f3b8d 2023-03-19 09:55:08,667 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/94edf94645b34afa9ad95a1bf748a9a2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/94edf94645b34afa9ad95a1bf748a9a2 2023-03-19 09:55:08,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c5d865bf5a83489a8cd85475fe4935e4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/c5d865bf5a83489a8cd85475fe4935e4 2023-03-19 09:55:08,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/d225887b4d9e45dc956959d255032554 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/d225887b4d9e45dc956959d255032554 2023-03-19 09:55:08,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/516c084a6cbe4a0f8a4c4c424a2f9800 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/516c084a6cbe4a0f8a4c4c424a2f9800 2023-03-19 09:55:08,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/589d53e2fc81420a81836eacfb5b8a53 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/589d53e2fc81420a81836eacfb5b8a53 2023-03-19 09:55:08,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8d386767b4674aafbbc202c5fdd7e915 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/8d386767b4674aafbbc202c5fdd7e915 2023-03-19 09:55:08,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f9c017585e094344a40bee2acc5f9a7c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/f9c017585e094344a40bee2acc5f9a7c 2023-03-19 09:55:08,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b66a060c5acd4ebd829340feebbb6e03 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/b66a060c5acd4ebd829340feebbb6e03 2023-03-19 09:55:08,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/653a2584f2e541528070559ee6f6f3fe to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/653a2584f2e541528070559ee6f6f3fe 2023-03-19 09:55:08,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7a370eb1b0094501aa55bf2f686b8c84 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7a370eb1b0094501aa55bf2f686b8c84 2023-03-19 09:55:08,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7b29dd539fd8476e9b5ae7794951f26e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/7b29dd539fd8476e9b5ae7794951f26e 2023-03-19 09:55:08,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/410bc6e72dbe4c7fbc6d409ed24b9390 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/410bc6e72dbe4c7fbc6d409ed24b9390 2023-03-19 09:55:08,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/036c86a08e33434489312aad5b325940 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/036c86a08e33434489312aad5b325940 2023-03-19 09:55:08,677 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/ae66dd0dbfe74afab45c799839dff645 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/ae66dd0dbfe74afab45c799839dff645 2023-03-19 09:55:08,678 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/cdfe179b889b43b08f7fd5114bbddcf3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/cdfe179b889b43b08f7fd5114bbddcf3 2023-03-19 09:55:08,679 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/0dcf61aaab3c4010a6519adffa291a1c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/0dcf61aaab3c4010a6519adffa291a1c 2023-03-19 09:55:08,682 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e10f643c3554d239247a4d131579751, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b3f84ed1aafb42889330a87c0eb723cb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/f25ab462e30d420bbc68dbd7d58488fd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ad1aa295760d45c38aa4922a07f46e6e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ace63dd026204c6ea6aff39bca1af86c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a2ecc67d45aa4078a106bbdad659e8e0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8a790cbed1774b39be41d1434d4604f9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/88c30d8d5538405596c5d8b95bb8e4c8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/83b23e2b930b4522a7d36d5b8b8346d7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6b2a20c90cdd431999f195222f5f2854, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a02038ca0b2b470e918eb8f8a6b4dcff, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2940cba211164ba391270663a37e6857, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/493d760fdbba468cb4f8c005914eb9cd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/3b3fcff14d3346d1964b889b5bb3b9c7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/e18446c0bb09483c80e8ccde2801a6c8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e4dc07390184322abd04fdc80452627, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/d254c1353b0e4539bcf288dfd3cb4d8b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1f7f03f4dc3d437aa4e8a69c45104a28, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a76afd45dd2c413e95d32aeaa21e0471, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/16e42ca82ba644acbfa1e4c9a9e87541, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/346ec05d7c7443328b6400f2ef376a82, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6402ccc8cc51453f8335685efb73602f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8b9a6fd8c5d24108aea96098f105dbba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c43c385127434c9f93fba47e66cf4f8f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1479a193bdd147f5bc446b2e683e1de5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/babd198595a44f53b42a4c2ee94acc70, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/546cfb9ff841419a898970fff20bd393, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/7077ac40fa0740bab1fb1776e4480ad5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/85b2561cd60d4e8b9825f449c7609bdb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b994f2f3ec304c5880916ffe621de725, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a0c6302d14f34d37a1c190e5b52b2b25, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/4fc0c79282aa4fd4b151ac3fadc8b085, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c83f711dbae840c7869aeda0a524c925, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c04b3fdc78c0479e9d926c56592cf00d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2746306c3a844b6081259ea0256ec001, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2c3a9b063fe24bc1b411d13d224be994, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ab42d2e31e184348ba2d26b68fe3ed0c] to archive 2023-03-19 09:55:08,683 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:55:08,684 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e10f643c3554d239247a4d131579751 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e10f643c3554d239247a4d131579751 2023-03-19 09:55:08,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b3f84ed1aafb42889330a87c0eb723cb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b3f84ed1aafb42889330a87c0eb723cb 2023-03-19 09:55:08,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/f25ab462e30d420bbc68dbd7d58488fd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/f25ab462e30d420bbc68dbd7d58488fd 2023-03-19 09:55:08,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ad1aa295760d45c38aa4922a07f46e6e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ad1aa295760d45c38aa4922a07f46e6e 2023-03-19 09:55:08,687 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ace63dd026204c6ea6aff39bca1af86c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ace63dd026204c6ea6aff39bca1af86c 2023-03-19 09:55:08,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a2ecc67d45aa4078a106bbdad659e8e0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a2ecc67d45aa4078a106bbdad659e8e0 2023-03-19 09:55:08,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8a790cbed1774b39be41d1434d4604f9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8a790cbed1774b39be41d1434d4604f9 2023-03-19 09:55:08,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/88c30d8d5538405596c5d8b95bb8e4c8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/88c30d8d5538405596c5d8b95bb8e4c8 2023-03-19 09:55:08,690 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/83b23e2b930b4522a7d36d5b8b8346d7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/83b23e2b930b4522a7d36d5b8b8346d7 2023-03-19 09:55:08,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6b2a20c90cdd431999f195222f5f2854 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6b2a20c90cdd431999f195222f5f2854 2023-03-19 09:55:08,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a02038ca0b2b470e918eb8f8a6b4dcff to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a02038ca0b2b470e918eb8f8a6b4dcff 2023-03-19 09:55:08,692 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2940cba211164ba391270663a37e6857 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2940cba211164ba391270663a37e6857 2023-03-19 09:55:08,693 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/493d760fdbba468cb4f8c005914eb9cd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/493d760fdbba468cb4f8c005914eb9cd 2023-03-19 09:55:08,694 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/3b3fcff14d3346d1964b889b5bb3b9c7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/3b3fcff14d3346d1964b889b5bb3b9c7 2023-03-19 09:55:08,695 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/e18446c0bb09483c80e8ccde2801a6c8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/e18446c0bb09483c80e8ccde2801a6c8 2023-03-19 09:55:08,695 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e4dc07390184322abd04fdc80452627 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1e4dc07390184322abd04fdc80452627 2023-03-19 09:55:08,696 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/d254c1353b0e4539bcf288dfd3cb4d8b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/d254c1353b0e4539bcf288dfd3cb4d8b 2023-03-19 09:55:08,697 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1f7f03f4dc3d437aa4e8a69c45104a28 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1f7f03f4dc3d437aa4e8a69c45104a28 2023-03-19 09:55:08,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a76afd45dd2c413e95d32aeaa21e0471 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a76afd45dd2c413e95d32aeaa21e0471 2023-03-19 09:55:08,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/16e42ca82ba644acbfa1e4c9a9e87541 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/16e42ca82ba644acbfa1e4c9a9e87541 2023-03-19 09:55:08,699 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/346ec05d7c7443328b6400f2ef376a82 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/346ec05d7c7443328b6400f2ef376a82 2023-03-19 09:55:08,700 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6402ccc8cc51453f8335685efb73602f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/6402ccc8cc51453f8335685efb73602f 2023-03-19 09:55:08,701 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8b9a6fd8c5d24108aea96098f105dbba to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/8b9a6fd8c5d24108aea96098f105dbba 2023-03-19 09:55:08,702 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c43c385127434c9f93fba47e66cf4f8f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c43c385127434c9f93fba47e66cf4f8f 2023-03-19 09:55:08,702 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1479a193bdd147f5bc446b2e683e1de5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/1479a193bdd147f5bc446b2e683e1de5 2023-03-19 09:55:08,703 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/babd198595a44f53b42a4c2ee94acc70 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/babd198595a44f53b42a4c2ee94acc70 2023-03-19 09:55:08,704 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2ebb8bf5b1024eb1bc7edc2ad5685ce5 2023-03-19 09:55:08,705 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/546cfb9ff841419a898970fff20bd393 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/546cfb9ff841419a898970fff20bd393 2023-03-19 09:55:08,706 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/7077ac40fa0740bab1fb1776e4480ad5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/7077ac40fa0740bab1fb1776e4480ad5 2023-03-19 09:55:08,707 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/85b2561cd60d4e8b9825f449c7609bdb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/85b2561cd60d4e8b9825f449c7609bdb 2023-03-19 09:55:08,709 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b994f2f3ec304c5880916ffe621de725 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/b994f2f3ec304c5880916ffe621de725 2023-03-19 09:55:08,710 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a0c6302d14f34d37a1c190e5b52b2b25 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/a0c6302d14f34d37a1c190e5b52b2b25 2023-03-19 09:55:08,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/4fc0c79282aa4fd4b151ac3fadc8b085 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/4fc0c79282aa4fd4b151ac3fadc8b085 2023-03-19 09:55:08,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c83f711dbae840c7869aeda0a524c925 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c83f711dbae840c7869aeda0a524c925 2023-03-19 09:55:08,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c04b3fdc78c0479e9d926c56592cf00d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/c04b3fdc78c0479e9d926c56592cf00d 2023-03-19 09:55:08,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2746306c3a844b6081259ea0256ec001 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2746306c3a844b6081259ea0256ec001 2023-03-19 09:55:08,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2c3a9b063fe24bc1b411d13d224be994 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2c3a9b063fe24bc1b411d13d224be994 2023-03-19 09:55:08,716 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ab42d2e31e184348ba2d26b68fe3ed0c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/ab42d2e31e184348ba2d26b68fe3ed0c 2023-03-19 09:55:08,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/41528741ba0d498b8cf1974ab3d2dbaf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/8a9048b06612443db0f59f1a22aa7cfc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/e5ec305e09f04f439d805d9d56b525a4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d3761806ba5942e383e5954f5153e424, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6e339cb2942b4939b625c1318d48471f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d808f211a6a2405d9be1311f78f14ada, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4f4818d2aa3f49d6bced9544e98fc298, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f3200995ac784716aa909850a497d7c7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4bb86a9f62e04458803b1090572978ea, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/62ca9158033e4d28aa1ff2ac808e8fce, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/43ab04ed702e438baf40dd96e6fe1a23, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/966f0042494a4599b16d990243a00cb7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/2172e02e22b440759b0243695144afb1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/0040728685154f848bd064631c2ccf13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/cd9e5ac6ee6a4c228f06025633f2817e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/bd818daa7d194af9a63876df59e9cc8c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe668b971d624c4db614f92e7ebcac46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b0cee81b47f14edbbf50e3cdf5515fcc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c0fe671e9e4b4bddaf2a63e02d8ab8af, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b5ab37b6820c44e390121d5247ee3ff3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/db8ed20c5be944ff9bd3e85a28f59c83, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/9ad49f3562f14d3680575ef590d7f7b1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/114148a181854621bfe5e6dc4d95e4a0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6c55f107d33949f38e49f511c79036c4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/66559e4f72e54e00ae8ed47cdf5b5769, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c6541b8544ff4b42970aca65a15c50c7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/300551f752b34f92b055f8a406e79291, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe303c5cac3440a4b7ec632e6235e595, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b9fb033c87444e0a83c3079176540ea8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c15fcdade2a84534974e06c2ba007ce1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c93eca3e8f494a5b8d99d1da9dc7271e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/1183eb1522d2487b8991e94ff2ec8749, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/7643c3f7feaf43949237ef3c5dec48d5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3cfc42e734524ad2892ae622806e5e63, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/57776c9bd38a4236842a110600aa59a6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/08f2cdc744e14b14813586947e1ecc63, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3c23a7e57f47476889e644cd1103dd79] to archive 2023-03-19 09:55:08,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:55:08,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/41528741ba0d498b8cf1974ab3d2dbaf to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/41528741ba0d498b8cf1974ab3d2dbaf 2023-03-19 09:55:08,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/8a9048b06612443db0f59f1a22aa7cfc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/8a9048b06612443db0f59f1a22aa7cfc 2023-03-19 09:55:08,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/e5ec305e09f04f439d805d9d56b525a4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/e5ec305e09f04f439d805d9d56b525a4 2023-03-19 09:55:08,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d3761806ba5942e383e5954f5153e424 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d3761806ba5942e383e5954f5153e424 2023-03-19 09:55:08,725 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6e339cb2942b4939b625c1318d48471f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6e339cb2942b4939b625c1318d48471f 2023-03-19 09:55:08,726 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d808f211a6a2405d9be1311f78f14ada to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/d808f211a6a2405d9be1311f78f14ada 2023-03-19 09:55:08,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4f4818d2aa3f49d6bced9544e98fc298 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4f4818d2aa3f49d6bced9544e98fc298 2023-03-19 09:55:08,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f3200995ac784716aa909850a497d7c7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f3200995ac784716aa909850a497d7c7 2023-03-19 09:55:08,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4bb86a9f62e04458803b1090572978ea to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/4bb86a9f62e04458803b1090572978ea 2023-03-19 09:55:08,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/62ca9158033e4d28aa1ff2ac808e8fce to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/62ca9158033e4d28aa1ff2ac808e8fce 2023-03-19 09:55:08,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/43ab04ed702e438baf40dd96e6fe1a23 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/43ab04ed702e438baf40dd96e6fe1a23 2023-03-19 09:55:08,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/966f0042494a4599b16d990243a00cb7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/966f0042494a4599b16d990243a00cb7 2023-03-19 09:55:08,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/2172e02e22b440759b0243695144afb1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/2172e02e22b440759b0243695144afb1 2023-03-19 09:55:08,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/0040728685154f848bd064631c2ccf13 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/0040728685154f848bd064631c2ccf13 2023-03-19 09:55:08,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/cd9e5ac6ee6a4c228f06025633f2817e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/cd9e5ac6ee6a4c228f06025633f2817e 2023-03-19 09:55:08,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/bd818daa7d194af9a63876df59e9cc8c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/bd818daa7d194af9a63876df59e9cc8c 2023-03-19 09:55:08,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe668b971d624c4db614f92e7ebcac46 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe668b971d624c4db614f92e7ebcac46 2023-03-19 09:55:08,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b0cee81b47f14edbbf50e3cdf5515fcc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b0cee81b47f14edbbf50e3cdf5515fcc 2023-03-19 09:55:08,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c0fe671e9e4b4bddaf2a63e02d8ab8af to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c0fe671e9e4b4bddaf2a63e02d8ab8af 2023-03-19 09:55:08,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b5ab37b6820c44e390121d5247ee3ff3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b5ab37b6820c44e390121d5247ee3ff3 2023-03-19 09:55:08,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/db8ed20c5be944ff9bd3e85a28f59c83 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/db8ed20c5be944ff9bd3e85a28f59c83 2023-03-19 09:55:08,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/9ad49f3562f14d3680575ef590d7f7b1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/9ad49f3562f14d3680575ef590d7f7b1 2023-03-19 09:55:08,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/114148a181854621bfe5e6dc4d95e4a0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/114148a181854621bfe5e6dc4d95e4a0 2023-03-19 09:55:08,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6c55f107d33949f38e49f511c79036c4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/6c55f107d33949f38e49f511c79036c4 2023-03-19 09:55:08,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/66559e4f72e54e00ae8ed47cdf5b5769 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/66559e4f72e54e00ae8ed47cdf5b5769 2023-03-19 09:55:08,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c6541b8544ff4b42970aca65a15c50c7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c6541b8544ff4b42970aca65a15c50c7 2023-03-19 09:55:08,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/300551f752b34f92b055f8a406e79291 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/300551f752b34f92b055f8a406e79291 2023-03-19 09:55:08,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe303c5cac3440a4b7ec632e6235e595 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/fe303c5cac3440a4b7ec632e6235e595 2023-03-19 09:55:08,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b9fb033c87444e0a83c3079176540ea8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/b9fb033c87444e0a83c3079176540ea8 2023-03-19 09:55:08,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c15fcdade2a84534974e06c2ba007ce1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c15fcdade2a84534974e06c2ba007ce1 2023-03-19 09:55:08,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c93eca3e8f494a5b8d99d1da9dc7271e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/c93eca3e8f494a5b8d99d1da9dc7271e 2023-03-19 09:55:08,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/1183eb1522d2487b8991e94ff2ec8749 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/1183eb1522d2487b8991e94ff2ec8749 2023-03-19 09:55:08,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/7643c3f7feaf43949237ef3c5dec48d5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/7643c3f7feaf43949237ef3c5dec48d5 2023-03-19 09:55:08,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3cfc42e734524ad2892ae622806e5e63 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3cfc42e734524ad2892ae622806e5e63 2023-03-19 09:55:08,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/57776c9bd38a4236842a110600aa59a6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/57776c9bd38a4236842a110600aa59a6 2023-03-19 09:55:08,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/08f2cdc744e14b14813586947e1ecc63 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/08f2cdc744e14b14813586947e1ecc63 2023-03-19 09:55:08,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3c23a7e57f47476889e644cd1103dd79 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/3c23a7e57f47476889e644cd1103dd79 2023-03-19 09:55:08,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/recovered.edits/909.seqid, newMaxSeqId=909, maxSeqId=1 2023-03-19 09:55:08,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0. 2023-03-19 09:55:08,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8dad03d10fa8d87944de8392ea9480f0: 2023-03-19 09:55:08,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:08,764 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=8dad03d10fa8d87944de8392ea9480f0, regionState=CLOSED 2023-03-19 09:55:08,764 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219708764"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219708764"}]},"ts":"1679219708764"} 2023-03-19 09:55:08,766 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-19 09:55:08,766 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 8dad03d10fa8d87944de8392ea9480f0, server=jenkins-hbase4.apache.org,34669,1679219618625 in 2.6030 sec 2023-03-19 09:55:08,768 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-19 09:55:08,768 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dad03d10fa8d87944de8392ea9480f0, UNASSIGN in 2.6070 sec 2023-03-19 09:55:08,768 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219708768"}]},"ts":"1679219708768"} 2023-03-19 09:55:08,769 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-19 09:55:08,770 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-19 09:55:08,772 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6170 sec 2023-03-19 09:55:10,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-19 09:55:10,262 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-19 09:55:10,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-19 09:55:10,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:10,264 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:10,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-19 09:55:10,265 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:10,269 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:10,270 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/recovered.edits] 2023-03-19 09:55:10,273 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/6e85db48590f4d3c916c44e199810a60 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/6e85db48590f4d3c916c44e199810a60 2023-03-19 09:55:10,274 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9b8d84eb98fe4fc7a24edaefc6c2da6f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/9b8d84eb98fe4fc7a24edaefc6c2da6f 2023-03-19 09:55:10,275 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dabcb1c6e4354dcf8a32ce5de17b8899 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/A/dabcb1c6e4354dcf8a32ce5de17b8899 2023-03-19 09:55:10,277 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2db901d57cc24005b354ba4bd9fc9e66 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/2db901d57cc24005b354ba4bd9fc9e66 2023-03-19 09:55:10,278 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/399837b847884b6f9a1275400caa4a4e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/399837b847884b6f9a1275400caa4a4e 2023-03-19 09:55:10,279 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/fbf1f686517b4564b5caac6edef2d095 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/B/fbf1f686517b4564b5caac6edef2d095 2023-03-19 09:55:10,281 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/01811017af4f45fdaa6e7010971186a0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/01811017af4f45fdaa6e7010971186a0 2023-03-19 09:55:10,282 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/84aba8cb46414ac1ab51e83d6aba7510 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/84aba8cb46414ac1ab51e83d6aba7510 2023-03-19 09:55:10,283 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f05a5ca514084ce1b26c36b8194d6624 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/C/f05a5ca514084ce1b26c36b8194d6624 2023-03-19 09:55:10,285 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/recovered.edits/909.seqid to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0/recovered.edits/909.seqid 2023-03-19 09:55:10,286 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/8dad03d10fa8d87944de8392ea9480f0 2023-03-19 09:55:10,286 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:55:10,288 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:10,289 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-19 09:55:10,290 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-19 09:55:10,291 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:10,291 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-19 09:55:10,291 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679219710291"}]},"ts":"9223372036854775807"} 2023-03-19 09:55:10,293 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-19 09:55:10,293 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8dad03d10fa8d87944de8392ea9480f0, NAME => 'TestAcidGuarantees,,1679219683677.8dad03d10fa8d87944de8392ea9480f0.', STARTKEY => '', ENDKEY => ''}] 2023-03-19 09:55:10,293 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-19 09:55:10,293 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679219710293"}]},"ts":"9223372036854775807"} 2023-03-19 09:55:10,297 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-19 09:55:10,299 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:10,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 37 msec 2023-03-19 09:55:10,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-19 09:55:10,366 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-19 09:55:10,377 INFO [Listener at localhost/45961] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 227), OpenFileDescriptor=559 (was 559), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 429), ProcessCount=169 (was 169), AvailableMemoryMB=1536 (was 1625) 2023-03-19 09:55:10,388 INFO [Listener at localhost/45961] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=169, AvailableMemoryMB=1536 2023-03-19 09:55:10,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:55:10,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:55:10,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:10,392 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-19 09:55:10,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-19 09:55:10,392 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-19 09:55:10,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-19 09:55:10,393 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:10,394 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 empty. 2023-03-19 09:55:10,394 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:10,394 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:55:10,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-19 09:55:10,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-19 09:55:10,800 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:55:10,802 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 72808d88f0cb7335353d6abe6e889fc0, NAME => 'TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp 2023-03-19 09:55:10,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-19 09:55:11,208 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:55:11,208 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 72808d88f0cb7335353d6abe6e889fc0, disabling compactions & flushes 2023-03-19 09:55:11,208 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,208 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,208 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. after waiting 0 ms 2023-03-19 09:55:11,208 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,208 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,208 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:11,210 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-19 09:55:11,211 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219711211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219711211"}]},"ts":"1679219711211"} 2023-03-19 09:55:11,212 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-19 09:55:11,212 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-19 09:55:11,213 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219711213"}]},"ts":"1679219711213"} 2023-03-19 09:55:11,213 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-19 09:55:11,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, ASSIGN}] 2023-03-19 09:55:11,217 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, ASSIGN 2023-03-19 09:55:11,218 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=false 2023-03-19 09:55:11,369 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:11,369 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219711369"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219711369"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219711369"}]},"ts":"1679219711369"} 2023-03-19 09:55:11,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:55:11,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-19 09:55:11,525 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 72808d88f0cb7335353d6abe6e889fc0, NAME => 'TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:55:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:55:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,526 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,527 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A 2023-03-19 09:55:11,527 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A 2023-03-19 09:55:11,528 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:11,528 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72808d88f0cb7335353d6abe6e889fc0 columnFamilyName A 2023-03-19 09:55:11,528 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(310): Store=72808d88f0cb7335353d6abe6e889fc0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:11,528 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,529 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B 2023-03-19 09:55:11,530 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B 2023-03-19 09:55:11,530 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:11,530 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72808d88f0cb7335353d6abe6e889fc0 columnFamilyName B 2023-03-19 09:55:11,530 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(310): Store=72808d88f0cb7335353d6abe6e889fc0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:11,530 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,532 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C 2023-03-19 09:55:11,532 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C 2023-03-19 09:55:11,532 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:11,532 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72808d88f0cb7335353d6abe6e889fc0 columnFamilyName C 2023-03-19 09:55:11,532 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(310): Store=72808d88f0cb7335353d6abe6e889fc0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:11,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:55:11,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:11,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:55:11,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 72808d88f0cb7335353d6abe6e889fc0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66590503, jitterRate=-0.007724180817604065}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:55:11,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:11,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., pid=37, masterSystemTime=1679219711522 2023-03-19 09:55:11,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:11,540 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:11,541 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219711540"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219711540"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219711540"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219711540"}]},"ts":"1679219711540"} 2023-03-19 09:55:11,542 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-19 09:55:11,542 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 in 171 msec 2023-03-19 09:55:11,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-19 09:55:11,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, ASSIGN in 326 msec 2023-03-19 09:55:11,544 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-19 09:55:11,544 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219711544"}]},"ts":"1679219711544"} 2023-03-19 09:55:11,545 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-19 09:55:11,547 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-19 09:55:11,547 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-03-19 09:55:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-19 09:55:12,497 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-19 09:55:12,497 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:12,508 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:12,509 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:12,511 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:12,512 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-19 09:55:12,513 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39924, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-19 09:55:12,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:55:12,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:55:12,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:12,926 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-19 09:55:12,927 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:55:12,927 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-19 09:55:12,929 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-19 09:55:12,932 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, REOPEN/MOVE}] 2023-03-19 09:55:12,932 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, REOPEN/MOVE 2023-03-19 09:55:12,933 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:12,933 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219712933"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219712933"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219712933"}]},"ts":"1679219712933"} 2023-03-19 09:55:12,934 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:55:13,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 72808d88f0cb7335353d6abe6e889fc0, disabling compactions & flushes 2023-03-19 09:55:13,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. after waiting 0 ms 2023-03-19 09:55:13,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-19 09:55:13,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:13,090 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 72808d88f0cb7335353d6abe6e889fc0 to self. 2023-03-19 09:55:13,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,092 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=CLOSED 2023-03-19 09:55:13,092 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219713092"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219713092"}]},"ts":"1679219713092"} 2023-03-19 09:55:13,094 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-19 09:55:13,094 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 in 158 msec 2023-03-19 09:55:13,094 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=true 2023-03-19 09:55:13,245 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,245 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219713244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219713244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219713244"}]},"ts":"1679219713244"} 2023-03-19 09:55:13,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:55:13,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 72808d88f0cb7335353d6abe6e889fc0, NAME => 'TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:55:13,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:55:13,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,402 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,403 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A 2023-03-19 09:55:13,403 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A 2023-03-19 09:55:13,403 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:13,403 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72808d88f0cb7335353d6abe6e889fc0 columnFamilyName A 2023-03-19 09:55:13,405 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(310): Store=72808d88f0cb7335353d6abe6e889fc0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:13,405 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,405 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B 2023-03-19 09:55:13,406 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B 2023-03-19 09:55:13,406 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:13,406 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72808d88f0cb7335353d6abe6e889fc0 columnFamilyName B 2023-03-19 09:55:13,406 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(310): Store=72808d88f0cb7335353d6abe6e889fc0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:13,406 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,407 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C 2023-03-19 09:55:13,407 DEBUG [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C 2023-03-19 09:55:13,407 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:13,407 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72808d88f0cb7335353d6abe6e889fc0 columnFamilyName C 2023-03-19 09:55:13,408 INFO [StoreOpener-72808d88f0cb7335353d6abe6e889fc0-1] regionserver.HStore(310): Store=72808d88f0cb7335353d6abe6e889fc0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:13,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:55:13,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 72808d88f0cb7335353d6abe6e889fc0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65001273, jitterRate=-0.031405553221702576}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:55:13,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:13,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., pid=42, masterSystemTime=1679219713397 2023-03-19 09:55:13,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,414 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,414 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219713414"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219713414"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219713414"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219713414"}]},"ts":"1679219713414"} 2023-03-19 09:55:13,416 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-19 09:55:13,416 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 in 169 msec 2023-03-19 09:55:13,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-19 09:55:13,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, REOPEN/MOVE in 484 msec 2023-03-19 09:55:13,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-19 09:55:13,419 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-03-19 09:55:13,420 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-03-19 09:55:13,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-19 09:55:13,421 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,429 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,429 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,435 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,436 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,439 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,439 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,442 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,443 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,446 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,446 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,450 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,451 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,454 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,454 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,460 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,460 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,464 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,464 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x0eb8f142 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:13,468 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e079da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:13,473 DEBUG [hconnection-0x34a87f8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,473 DEBUG [hconnection-0x4fe62ced-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,474 DEBUG [hconnection-0x671a1f27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,475 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,475 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46810, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:13,476 DEBUG [hconnection-0x7dc1d05e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,476 DEBUG [hconnection-0x19ebacea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,476 DEBUG [hconnection-0x5bc332cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,480 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46824, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,480 DEBUG [hconnection-0x16fe3ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,481 DEBUG [hconnection-0x622b076c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,481 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46862, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,483 DEBUG [hconnection-0x99b9c36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,485 DEBUG [hconnection-0x57f7e6b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:13,486 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46880, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,489 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46834, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,489 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46818, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,489 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46864, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,489 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46848, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,490 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46876, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:13,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:13,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:13,494 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:13,494 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:13,494 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:13,494 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:13,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:13,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:13,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:13,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:13,496 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:13,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:13,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:13,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:13,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:13,498 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:13,498 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:13,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:13,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:13,500 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:13,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219773521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219773522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219773522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219773523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219773523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219773624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219773624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219773624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219773625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219773625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219773829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219773829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219773829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219773829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:13,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219773830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:13,938 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303195d63d30ab2cc4d1e9a90c02818a9939f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195d63d30ab2cc4d1e9a90c02818a9939f 2023-03-19 09:55:14,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219774133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219774134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219774134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219774135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219774135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,297 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-19 09:55:14,344 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/2f29efcec2bd410ab7c3efeb0636e0a3 2023-03-19 09:55:14,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219774641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219774641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219774643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219774644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:14,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219774644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:14,755 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9 2023-03-19 09:55:15,168 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/c91ea9c716df4c12a9afd0fb11e59b55 2023-03-19 09:55:15,182 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/2f29efcec2bd410ab7c3efeb0636e0a3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f29efcec2bd410ab7c3efeb0636e0a3 2023-03-19 09:55:15,189 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f29efcec2bd410ab7c3efeb0636e0a3, entries=150, sequenceid=17, filesize=25.1 K 2023-03-19 09:55:15,189 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9 2023-03-19 09:55:15,196 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9, entries=150, sequenceid=17, filesize=11.6 K 2023-03-19 09:55:15,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/c91ea9c716df4c12a9afd0fb11e59b55 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c91ea9c716df4c12a9afd0fb11e59b55 2023-03-19 09:55:15,203 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c91ea9c716df4c12a9afd0fb11e59b55, entries=150, sequenceid=17, filesize=11.6 K 2023-03-19 09:55:15,204 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 72808d88f0cb7335353d6abe6e889fc0 in 1704ms, sequenceid=17, compaction requested=false 2023-03-19 09:55:15,204 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-19 09:55:15,204 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:15,204 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:15,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:15,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:15,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:15,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:15,206 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:15,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:15,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2516a757[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:15,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:15,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:15,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:15,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:15,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:15,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:15,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:15,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:15,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:15,214 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:15,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:15,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:15,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:15,215 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:15,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:15,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:15,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:15,218 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:15,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:15,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:15,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:15,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:15,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:15,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:15,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:15,224 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:15,224 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:15,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:15,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:15,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:15,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:15,324 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:15,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:15,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:15,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:15,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:15,327 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:15,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:15,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:15,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:15,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:15,329 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:15,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:15,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:15,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:15,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:15,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:15,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:15,330 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:15,330 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:15,330 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:15,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:15,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:15,331 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:15,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:15,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:15,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:15,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219775660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219775661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219775662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219775663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219775664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,749 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319694399355ff144f6b09d20eba5137a8a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319694399355ff144f6b09d20eba5137a8a 2023-03-19 09:55:15,759 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/7edca62894084a01b9d3126d9721cf64 2023-03-19 09:55:15,768 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/877b817f7c9f479b9d693cd9d18e8dc7 2023-03-19 09:55:15,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219775770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219775770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219775770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219775772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219775772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219775972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219775972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219775972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219775975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:15,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:15,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219775976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,180 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/4c42768b47bb442b80edf39e3f70da85 2023-03-19 09:55:16,184 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/7edca62894084a01b9d3126d9721cf64 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7edca62894084a01b9d3126d9721cf64 2023-03-19 09:55:16,187 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7edca62894084a01b9d3126d9721cf64, entries=150, sequenceid=37, filesize=25.1 K 2023-03-19 09:55:16,188 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/877b817f7c9f479b9d693cd9d18e8dc7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/877b817f7c9f479b9d693cd9d18e8dc7 2023-03-19 09:55:16,191 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/877b817f7c9f479b9d693cd9d18e8dc7, entries=150, sequenceid=37, filesize=11.6 K 2023-03-19 09:55:16,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/4c42768b47bb442b80edf39e3f70da85 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4c42768b47bb442b80edf39e3f70da85 2023-03-19 09:55:16,195 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4c42768b47bb442b80edf39e3f70da85, entries=150, sequenceid=37, filesize=11.6 K 2023-03-19 09:55:16,196 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 72808d88f0cb7335353d6abe6e889fc0 in 865ms, sequenceid=37, compaction requested=false 2023-03-19 09:55:16,196 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:16,196 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:16,201 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,202 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,202 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:16,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:16,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b4e9af[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:16,204 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:16,204 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,205 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:16,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:16,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:16,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:16,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:16,210 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:16,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:16,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:16,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:16,210 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:16,211 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,211 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,211 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:16,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:16,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:16,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,217 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:16,217 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:16,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:16,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:55:16,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:16,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:16,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:16,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:16,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:16,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:16,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219776297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219776298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219776298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219776299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219776302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,317 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:16,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:16,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:16,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:16,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:16,320 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:16,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:16,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:16,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,321 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:16,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:16,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:16,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:16,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:16,323 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,323 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:16,323 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:16,323 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:16,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:16,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,325 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:16,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b2ff549[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:16,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:16,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:16,328 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:16,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,329 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:16,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:16,332 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:16,332 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:16,332 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:16,332 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:16,332 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:16,333 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,333 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,333 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,334 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,335 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:16,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:16,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,337 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,338 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:16,338 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:16,338 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,338 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,338 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219776404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219776409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219776409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219776409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219776409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,438 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:16,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:16,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:16,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:16,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:16,441 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:16,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:16,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:16,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,454 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,454 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,454 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:16,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:16,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:16,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:16,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:16,455 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:16,455 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,455 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:16,456 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:16,456 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:16,457 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,457 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:16,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55735d4c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:16,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:16,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,460 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,460 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:16,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:16,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:16,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:16,464 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:16,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:16,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:16,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:16,464 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:16,470 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,470 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,470 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:16,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:16,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:16,475 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:16,475 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:16,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,575 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:16,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:16,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:16,579 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:16,579 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,579 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:16,579 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:16,579 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,579 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,581 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,581 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,581 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,581 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:16,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:16,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:16,582 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,583 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:16,583 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:16,583 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:16,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:16,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:16,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:16,584 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,584 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,585 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,585 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:16,586 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:16,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454f74aa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:16,586 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,588 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:16,589 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,589 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:16,590 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,590 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,590 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,590 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:16,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:16,593 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:16,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:16,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:16,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:16,593 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:16,594 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,594 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,594 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:16,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,600 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,600 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:16,600 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:16,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219776609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219776613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219776614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219776614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219776615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,689 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319db8592f895e84ad7a5d9a9a660460296 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319db8592f895e84ad7a5d9a9a660460296 2023-03-19 09:55:16,701 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:16,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:16,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:16,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:16,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:16,705 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:16,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:16,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:16,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,707 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,707 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:16,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:16,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:16,710 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,710 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:16,710 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:16,710 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:16,710 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:16,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c1043d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:16,713 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:16,713 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,715 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:16,715 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:16,715 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:16,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:16,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:16,717 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:16,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:16,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:16,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:16,717 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:16,719 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,719 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,719 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,721 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:16,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:16,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,725 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:16,725 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:16,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:16,825 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:16,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:16,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:16,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:16,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:16,829 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:16,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:16,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:16,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:16,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:16,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:16,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:16,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:16,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:16,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:16,836 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,836 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:16,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:16,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6702a91[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:16,843 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:16,843 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,844 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:16,844 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:16,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:16,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:16,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:16,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:16,850 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:16,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:16,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:16,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:16,850 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:16,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,858 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:16,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:16,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:16,859 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:16,859 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:16,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219776916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219776920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219776920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219776920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:16,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219776924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,959 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:16,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:16,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:16,963 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:16,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:16,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:16,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,966 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,966 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:16,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:16,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:16,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:16,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:16,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:16,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:16,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:16,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:16,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:16,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:16,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:16,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a69e2e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:16,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,974 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:16,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:16,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:16,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:16,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:16,978 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:16,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:16,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:16,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:16,978 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:16,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:16,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,981 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:16,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:16,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:16,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,985 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:16,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:16,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:16,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:16,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:16,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:16,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:16,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:16,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:16,985 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:16,985 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:16,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:16,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,085 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:17,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:17,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:17,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:17,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:17,092 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:17,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:17,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:17,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,095 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,095 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,095 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/ae4a0cc2a2bb4b139f429e33318b8706 2023-03-19 09:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:17,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:17,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:17,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:17,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:17,097 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:17,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:17,098 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:17,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:17,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f24994[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:17,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:17,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:17,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:17,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:17,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:17,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:17,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:17,121 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:17,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:17,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:17,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:17,121 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:17,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,128 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,130 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,134 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,134 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,134 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,134 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,134 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:17,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:17,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,134 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:17,134 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:17,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/6b33d9430658402e832a13a736180394 2023-03-19 09:55:17,234 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:17,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:17,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:17,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:17,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:17,238 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:17,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:17,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:17,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,240 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:17,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:17,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:17,243 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,243 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:17,243 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:17,243 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:17,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:17,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:17,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:17,248 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:17,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da84bbb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:17,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:17,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:17,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:17,252 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:17,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,253 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:17,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:17,255 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:17,255 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:17,255 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:17,255 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:17,255 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:17,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,261 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,263 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:17,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:17,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,264 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:17,264 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:17,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:17,364 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:17,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:17,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:17,367 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:17,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,368 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:17,368 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:17,368 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,368 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,369 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,370 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:17,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:17,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:17,371 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,371 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:17,371 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:17,371 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:17,371 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:17,373 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,373 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,373 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,373 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:17,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a8bd43d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:17,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:17,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,376 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:17,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:17,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:17,377 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,377 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:17,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:17,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:17,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:17,381 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:17,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:17,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:17,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:17,381 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:17,382 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,382 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,382 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,383 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,386 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:17,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:17,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:17,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,387 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:17,387 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:17,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:17,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219777420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:17,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219777430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:17,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219777430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:17,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219777430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:17,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219777433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,487 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:17,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:17,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:17,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:17,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:17,490 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:17,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:17,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:17,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,493 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:17,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:17,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:17,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:17,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:17,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:17,502 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:17,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@372174ac[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:17,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:17,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:17,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:17,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:17,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:17,513 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:17,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:17,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:17,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:17,513 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:17,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:17,517 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:17,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:17,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:17,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:17,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:17,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:17,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:17,526 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:17,526 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:17,526 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:17,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:17,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/314ff6fe849147fb99676ccddb0a9e48 2023-03-19 09:55:17,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/ae4a0cc2a2bb4b139f429e33318b8706 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ae4a0cc2a2bb4b139f429e33318b8706 2023-03-19 09:55:17,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ae4a0cc2a2bb4b139f429e33318b8706, entries=200, sequenceid=50, filesize=31.9 K 2023-03-19 09:55:17,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/6b33d9430658402e832a13a736180394 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/6b33d9430658402e832a13a736180394 2023-03-19 09:55:17,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/6b33d9430658402e832a13a736180394, entries=150, sequenceid=50, filesize=11.6 K 2023-03-19 09:55:17,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/314ff6fe849147fb99676ccddb0a9e48 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/314ff6fe849147fb99676ccddb0a9e48 2023-03-19 09:55:17,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/314ff6fe849147fb99676ccddb0a9e48, entries=150, sequenceid=50, filesize=11.6 K 2023-03-19 09:55:17,587 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 72808d88f0cb7335353d6abe6e889fc0 in 1310ms, sequenceid=50, compaction requested=true 2023-03-19 09:55:17,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:17,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:17,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-19 09:55:17,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-19 09:55:17,589 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:17,590 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:17,591 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:17,591 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:17,591 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/A is initiating minor compaction (all files) 2023-03-19 09:55:17,591 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/C is initiating minor compaction (all files) 2023-03-19 09:55:17,591 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/A in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,591 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/C in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,591 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f29efcec2bd410ab7c3efeb0636e0a3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7edca62894084a01b9d3126d9721cf64, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ae4a0cc2a2bb4b139f429e33318b8706] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=82.2 K 2023-03-19 09:55:17,591 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c91ea9c716df4c12a9afd0fb11e59b55, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4c42768b47bb442b80edf39e3f70da85, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/314ff6fe849147fb99676ccddb0a9e48] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=34.8 K 2023-03-19 09:55:17,591 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2f29efcec2bd410ab7c3efeb0636e0a3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679219713497 2023-03-19 09:55:17,591 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c91ea9c716df4c12a9afd0fb11e59b55, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679219713497 2023-03-19 09:55:17,592 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7edca62894084a01b9d3126d9721cf64, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679219713516 2023-03-19 09:55:17,592 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4c42768b47bb442b80edf39e3f70da85, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679219713516 2023-03-19 09:55:17,592 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ae4a0cc2a2bb4b139f429e33318b8706, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219715661 2023-03-19 09:55:17,592 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 314ff6fe849147fb99676ccddb0a9e48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219715661 2023-03-19 09:55:17,626 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:17,630 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#C#compaction#328 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:17,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:17,658 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#A#compaction#327 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:17,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:17,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:17,668 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:17,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:17,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:17,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,669 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,670 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,670 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:17,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:17,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:17,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:17,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:17,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:17,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:17,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:17,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:17,692 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,693 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:17,693 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-19 09:55:17,693 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:17,693 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:17,694 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:17,694 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:17,694 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:17,694 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:17,700 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/943adc0fcc3f40219f9d7bb534d337ea as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/943adc0fcc3f40219f9d7bb534d337ea 2023-03-19 09:55:17,707 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/C of 72808d88f0cb7335353d6abe6e889fc0 into 943adc0fcc3f40219f9d7bb534d337ea(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:17,707 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:17,707 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/C, priority=13, startTime=1679219717589; duration=0sec 2023-03-19 09:55:17,707 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:17,707 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:17,709 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:17,709 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/B is initiating minor compaction (all files) 2023-03-19 09:55:17,709 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/B in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:17,709 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/877b817f7c9f479b9d693cd9d18e8dc7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/6b33d9430658402e832a13a736180394] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=34.8 K 2023-03-19 09:55:17,710 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 49e32ce7c0ba4e4f9d1b86aff0b98bc9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679219713497 2023-03-19 09:55:17,711 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 877b817f7c9f479b9d693cd9d18e8dc7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679219713516 2023-03-19 09:55:17,711 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6b33d9430658402e832a13a736180394, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219715661 2023-03-19 09:55:17,769 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#B#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:17,806 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303196f59f45e6ee04ffcb1f461e695714e89 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196f59f45e6ee04ffcb1f461e695714e89 2023-03-19 09:55:18,199 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/242d946e910b4dd3b6aff246d44ec31a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/242d946e910b4dd3b6aff246d44ec31a 2023-03-19 09:55:18,206 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/e91db938a24f4e4dad7d373eb1b3db4b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e91db938a24f4e4dad7d373eb1b3db4b 2023-03-19 09:55:18,207 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/A of 72808d88f0cb7335353d6abe6e889fc0 into 242d946e910b4dd3b6aff246d44ec31a(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:18,207 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:18,207 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/A, priority=13, startTime=1679219717589; duration=0sec 2023-03-19 09:55:18,207 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:18,211 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/B of 72808d88f0cb7335353d6abe6e889fc0 into e91db938a24f4e4dad7d373eb1b3db4b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:18,211 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:18,211 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/B, priority=13, startTime=1679219717589; duration=0sec 2023-03-19 09:55:18,211 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:18,243 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/53d48a0a0978445aaeeda13b1caca22e 2023-03-19 09:55:18,254 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/08a0fc7a4fff42e6b29c423e5a7d7e58 2023-03-19 09:55:18,299 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/f66d25ce6db8451f8a12490570f7e3f9 2023-03-19 09:55:18,304 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/53d48a0a0978445aaeeda13b1caca22e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/53d48a0a0978445aaeeda13b1caca22e 2023-03-19 09:55:18,308 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/53d48a0a0978445aaeeda13b1caca22e, entries=150, sequenceid=69, filesize=25.1 K 2023-03-19 09:55:18,309 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/08a0fc7a4fff42e6b29c423e5a7d7e58 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/08a0fc7a4fff42e6b29c423e5a7d7e58 2023-03-19 09:55:18,313 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/08a0fc7a4fff42e6b29c423e5a7d7e58, entries=150, sequenceid=69, filesize=11.6 K 2023-03-19 09:55:18,314 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/f66d25ce6db8451f8a12490570f7e3f9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f66d25ce6db8451f8a12490570f7e3f9 2023-03-19 09:55:18,322 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f66d25ce6db8451f8a12490570f7e3f9, entries=150, sequenceid=69, filesize=11.6 K 2023-03-19 09:55:18,322 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 72808d88f0cb7335353d6abe6e889fc0 in 629ms, sequenceid=69, compaction requested=false 2023-03-19 09:55:18,323 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:18,323 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:18,324 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527a2535[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:18,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:18,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,329 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:18,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,330 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:18,333 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,333 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:18,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:18,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:18,346 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:18,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:18,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:18,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:18,346 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:18,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,351 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:18,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:18,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:18,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:18,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,355 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:18,355 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:18,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:18,455 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:18,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:18,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:18,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:18,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:18,458 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:18,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:18,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:18,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:18,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:18,461 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:18,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:18,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:18,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:18,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:18,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:18,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:18,500 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:18,500 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:18,500 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-19 09:55:18,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:18,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:18,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:18,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:18,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:18,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:18,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:18,525 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:18,554 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319bc226da026c0475a8105eb5c0c489a2f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bc226da026c0475a8105eb5c0c489a2f 2023-03-19 09:55:18,560 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=77, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/fe4c955079244c7fbaebf538fa159e13 2023-03-19 09:55:18,574 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/b11fe9ea218e46a6bd0eee837797e3c5 2023-03-19 09:55:18,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679219718504, max timestamp=1679219718576Num uniques 859; 2023-03-19 09:55:18,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-19 09:55:18,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679219718504, max timestamp=1679219718576Num uniques 859; 2023-03-19 09:55:18,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679219718504, max timestamp=1679219718576Num uniques 859; 2023-03-19 09:55:18,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-19 09:55:18,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-19 09:55:18,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219778594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,610 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/0597f22f2bb44612be633c6b39f3f1b0 2023-03-19 09:55:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219778603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219778609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219778610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219778612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,624 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/fe4c955079244c7fbaebf538fa159e13 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/fe4c955079244c7fbaebf538fa159e13 2023-03-19 09:55:18,628 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/fe4c955079244c7fbaebf538fa159e13, entries=100, sequenceid=77, filesize=18.4 K 2023-03-19 09:55:18,629 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/b11fe9ea218e46a6bd0eee837797e3c5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b11fe9ea218e46a6bd0eee837797e3c5 2023-03-19 09:55:18,636 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b11fe9ea218e46a6bd0eee837797e3c5, entries=100, sequenceid=77, filesize=9.3 K 2023-03-19 09:55:18,637 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/0597f22f2bb44612be633c6b39f3f1b0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/0597f22f2bb44612be633c6b39f3f1b0 2023-03-19 09:55:18,643 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/0597f22f2bb44612be633c6b39f3f1b0, entries=100, sequenceid=77, filesize=9.3 K 2023-03-19 09:55:18,644 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=194.56 KB/199230 for 72808d88f0cb7335353d6abe6e889fc0 in 144ms, sequenceid=77, compaction requested=true 2023-03-19 09:55:18,644 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:18,644 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:18,644 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:18,644 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:18,644 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,644 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:18,644 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:18,646 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d005952[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:18,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:18,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,650 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,650 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:18,650 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:18,650 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:18,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:18,651 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,652 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:18,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:18,654 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:18,654 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:18,654 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:18,654 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:18,654 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:18,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:18,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:18,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:18,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,663 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:18,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:18,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:18,663 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:18,663 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:18,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:18,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=201.27 KB heapSize=518.58 KB 2023-03-19 09:55:18,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219778713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:18,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:18,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:18,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:18,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:18,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:18,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219778717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219778719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219778720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219778724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,738 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303191ca788eb48ad4766b2db97a465cb5b30 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303191ca788eb48ad4766b2db97a465cb5b30 2023-03-19 09:55:18,763 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:18,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:18,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:18,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:18,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:18,776 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:18,776 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:18,776 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:18,776 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:18,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,780 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:18,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:18,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:18,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:18,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:18,783 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:18,783 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:18,783 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:18,783 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:18,783 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:18,783 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:18,783 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:18,784 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:18,784 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:18,786 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:18,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49be44f3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:18,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:18,788 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,789 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:18,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:18,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:18,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:18,792 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,792 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:18,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:18,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:18,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:18,795 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:18,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:18,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:18,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:18,796 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:18,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,798 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,798 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:18,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:18,805 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:18,805 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:18,805 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:18,805 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,805 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:18,805 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,805 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,805 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,806 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,806 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,806 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,806 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:18,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219778828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,905 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:18,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:18,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:18,908 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:18,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:18,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:18,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:18,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,913 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:18,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:18,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,914 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:18,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:18,915 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:18,915 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:18,915 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:18,915 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,917 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:18,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7506cd7c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:18,919 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,920 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,920 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:18,920 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:18,920 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:18,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,921 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:18,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:18,924 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:18,924 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:18,924 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:18,924 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:18,924 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:18,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219778921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219778921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:18,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,934 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219778930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:18,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219778931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:18,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:18,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:18,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:18,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:18,943 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:18,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:18,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:18,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:18,943 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:18,943 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:18,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:18,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:18,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:18,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219779033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,044 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,047 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,049 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,049 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:19,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,050 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,050 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,050 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,050 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,051 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,052 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,052 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76560d64[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,056 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,057 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,058 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,060 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,060 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,060 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,060 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,060 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,062 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,062 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,064 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,067 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:19,067 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,067 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,154 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=67.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/e69be05be9af414ea5bec370d929ec71 2023-03-19 09:55:19,167 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,170 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,170 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,171 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,171 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:19,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,175 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,175 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,175 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,175 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,177 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c31035[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,181 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,182 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,182 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,186 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,186 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,189 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,189 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,189 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,191 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,192 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,194 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,194 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,194 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219779226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219779227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219779235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219779238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,294 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,298 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,300 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12614022[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,307 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,307 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,311 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,311 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,311 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,313 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,313 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,315 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,315 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,317 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,317 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,317 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,317 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:19,328 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,328 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219779336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,429 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,432 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,433 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:19,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,437 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cf123f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,445 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,445 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,445 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,445 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,445 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,446 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,446 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,446 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,447 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,450 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,450 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,450 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,551 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,554 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,555 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,555 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,556 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,558 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,558 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,558 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,558 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,562 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b4084f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,564 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,565 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/7e63dab504874ca2b1dc2f4137919efa 2023-03-19 09:55:19,565 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,566 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,569 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,569 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,572 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,572 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,572 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,574 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,582 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:19,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,583 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,583 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,683 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,686 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,687 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,687 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:19,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,690 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,691 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,691 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,691 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,692 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,693 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,693 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2647f034[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,695 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,699 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,700 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,704 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:19,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,708 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,709 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,709 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219779733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219779734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219779738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219779745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,809 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,812 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,815 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:19,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,817 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,817 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,817 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,817 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,819 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,819 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,819 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,819 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebfd48e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,821 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,821 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,822 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,826 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,826 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,836 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,836 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,836 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,836 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219779842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,936 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:19,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:19,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:19,942 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:19,942 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,942 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:19,942 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:19,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,945 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,945 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:19,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:19,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:19,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:19,947 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,947 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:19,947 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:19,947 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:19,953 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:19,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c8678c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:19,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:19,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,957 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:19,957 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,957 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:19,959 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:19,962 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:19,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:19,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:19,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:19,962 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:19,963 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,963 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,963 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:19,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:19,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:19,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:19,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:19,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:19,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:19,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:19,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:19,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:19,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:19,968 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:19,968 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:19,968 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:19,968 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:19,968 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:19,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/f746664af2974d64854abd3c1f67e79d 2023-03-19 09:55:19,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/e69be05be9af414ea5bec370d929ec71 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/e69be05be9af414ea5bec370d929ec71 2023-03-19 09:55:19,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/e69be05be9af414ea5bec370d929ec71, entries=200, sequenceid=110, filesize=31.9 K 2023-03-19 09:55:19,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/7e63dab504874ca2b1dc2f4137919efa as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/7e63dab504874ca2b1dc2f4137919efa 2023-03-19 09:55:19,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/7e63dab504874ca2b1dc2f4137919efa, entries=150, sequenceid=110, filesize=11.6 K 2023-03-19 09:55:19,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/f746664af2974d64854abd3c1f67e79d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f746664af2974d64854abd3c1f67e79d 2023-03-19 09:55:19,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f746664af2974d64854abd3c1f67e79d, entries=150, sequenceid=110, filesize=11.6 K 2023-03-19 09:55:19,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~201.27 KB/206100, heapSize ~518.53 KB/530976, currentSize=0 B/0 for 72808d88f0cb7335353d6abe6e889fc0 in 1278ms, sequenceid=110, compaction requested=true 2023-03-19 09:55:19,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:19,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:19,998 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:55:19,998 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:55:19,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:19,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:19,999 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103079 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:55:19,999 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/A is initiating minor compaction (all files) 2023-03-19 09:55:19,999 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:55:19,999 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/A in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,999 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/B is initiating minor compaction (all files) 2023-03-19 09:55:19,999 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/242d946e910b4dd3b6aff246d44ec31a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/53d48a0a0978445aaeeda13b1caca22e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/fe4c955079244c7fbaebf538fa159e13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/e69be05be9af414ea5bec370d929ec71] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=100.7 K 2023-03-19 09:55:19,999 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/B in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:19,999 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e91db938a24f4e4dad7d373eb1b3db4b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/08a0fc7a4fff42e6b29c423e5a7d7e58, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b11fe9ea218e46a6bd0eee837797e3c5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/7e63dab504874ca2b1dc2f4137919efa] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=44.2 K 2023-03-19 09:55:20,000 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 242d946e910b4dd3b6aff246d44ec31a, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219715661 2023-03-19 09:55:20,000 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e91db938a24f4e4dad7d373eb1b3db4b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219715661 2023-03-19 09:55:20,000 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 53d48a0a0978445aaeeda13b1caca22e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679219716296 2023-03-19 09:55:20,000 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 08a0fc7a4fff42e6b29c423e5a7d7e58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679219716296 2023-03-19 09:55:20,000 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting fe4c955079244c7fbaebf538fa159e13, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1679219718493 2023-03-19 09:55:20,000 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b11fe9ea218e46a6bd0eee837797e3c5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1679219718493 2023-03-19 09:55:20,001 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting e69be05be9af414ea5bec370d929ec71, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679219718606 2023-03-19 09:55:20,001 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 7e63dab504874ca2b1dc2f4137919efa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679219718606 2023-03-19 09:55:20,009 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#B#compaction#340 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:20,013 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#A#compaction#339 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:20,024 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/def4815270f8461f990ca77ca568e311 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/def4815270f8461f990ca77ca568e311 2023-03-19 09:55:20,028 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/B of 72808d88f0cb7335353d6abe6e889fc0 into def4815270f8461f990ca77ca568e311(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:20,028 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,028 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/B, priority=12, startTime=1679219719998; duration=0sec 2023-03-19 09:55:20,028 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:20,028 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:55:20,030 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:55:20,030 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/C is initiating minor compaction (all files) 2023-03-19 09:55:20,030 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/C in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,030 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/943adc0fcc3f40219f9d7bb534d337ea, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f66d25ce6db8451f8a12490570f7e3f9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/0597f22f2bb44612be633c6b39f3f1b0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f746664af2974d64854abd3c1f67e79d] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=44.2 K 2023-03-19 09:55:20,030 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 943adc0fcc3f40219f9d7bb534d337ea, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679219715661 2023-03-19 09:55:20,031 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f66d25ce6db8451f8a12490570f7e3f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679219716296 2023-03-19 09:55:20,031 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0597f22f2bb44612be633c6b39f3f1b0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1679219718493 2023-03-19 09:55:20,031 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f746664af2974d64854abd3c1f67e79d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679219718606 2023-03-19 09:55:20,050 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#C#compaction#341 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:20,068 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:20,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:20,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:20,071 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:20,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:20,072 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:20,072 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,072 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,075 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,075 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:20,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:20,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:20,076 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,076 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:20,076 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,077 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:20,077 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:20,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:20,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2089e34a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:20,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:20,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:20,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:20,083 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:20,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:20,086 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:20,086 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:20,086 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:20,086 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:20,086 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:20,088 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,088 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,089 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:20,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:20,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,093 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:20,093 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:20,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:20,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,094 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,094 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,193 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:20,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:20,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:20,199 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:20,199 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,199 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:20,199 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:20,199 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,199 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,201 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,201 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,201 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,201 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:20,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:20,203 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,203 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:20,203 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,203 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,209 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d6613f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:20,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:20,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:20,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:20,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:20,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:20,213 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:20,216 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:20,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:20,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:20,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:20,216 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:20,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,218 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:20,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,224 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:20,224 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:20,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,324 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:20,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:20,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:20,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:20,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:20,327 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:20,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:20,327 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:20,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,329 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,330 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,330 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:20,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:20,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:20,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,331 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:20,333 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,333 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,333 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,333 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:20,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790dd92f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:20,335 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:20,335 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,336 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:20,336 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,336 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:20,339 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,339 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,339 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,339 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:20,342 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:20,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:20,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:20,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:20,342 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:20,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,344 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:20,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:20,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:20,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,348 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:20,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,348 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:20,349 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,349 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,442 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/0ddd938ec64641c0ad65e1741961f77a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0ddd938ec64641c0ad65e1741961f77a 2023-03-19 09:55:20,445 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/A of 72808d88f0cb7335353d6abe6e889fc0 into 0ddd938ec64641c0ad65e1741961f77a(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:20,445 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,446 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/A, priority=12, startTime=1679219719998; duration=0sec 2023-03-19 09:55:20,446 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:20,449 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:20,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:20,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:20,456 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:20,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:20,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:20,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,458 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:20,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:20,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:20,460 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,460 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:20,460 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,460 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:20,463 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:20,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dad3585[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:20,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:20,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:20,469 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,469 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/94258e3146594ee39b338f284e00de02 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/94258e3146594ee39b338f284e00de02 2023-03-19 09:55:20,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:20,471 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,473 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/C of 72808d88f0cb7335353d6abe6e889fc0 into 94258e3146594ee39b338f284e00de02(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:20,473 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,473 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/C, priority=12, startTime=1679219719998; duration=0sec 2023-03-19 09:55:20,474 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:20,474 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:20,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:20,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:20,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:20,474 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:20,476 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,476 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,476 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,477 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,478 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,481 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:20,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,482 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:20,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,482 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:20,582 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:20,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:20,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:20,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:20,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:20,604 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:20,604 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,604 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:20,604 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:20,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,606 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:20,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:20,608 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,608 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:20,608 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,608 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:20,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d100c9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:20,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:20,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:20,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:20,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:20,620 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:20,620 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:20,620 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:20,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:20,621 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:20,622 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,622 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,622 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,623 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:20,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:20,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:20,627 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:20,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,627 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:20,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,727 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:20,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:20,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:20,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:20,731 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:20,731 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,731 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:20,731 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:20,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,733 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,734 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,734 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:20,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:20,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:20,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:20,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:20,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:20,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:20,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:20,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:20,736 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,736 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:20,736 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:20,736 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:20,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:20,740 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48844e35[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:20,740 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,742 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:20,742 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,742 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:20,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:20,746 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:20,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:20,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:20,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:20,747 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:20,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:20,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:20,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:20,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:20,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:20,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:20,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:20,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:20,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,755 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:20,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:20,755 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:20,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,855 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:20,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:20,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:20,859 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:20,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:20,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:20,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:20,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:20,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:20,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:20,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:20,864 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-19 09:55:20,864 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:20,864 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:20,864 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:20,864 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:20,864 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:20,864 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:20,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:20,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:20,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219720898, max timestamp=1679219720910Num uniques 859; 2023-03-19 09:55:20,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:55:20,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219720898, max timestamp=1679219720910Num uniques 859; 2023-03-19 09:55:20,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219720898, max timestamp=1679219720910Num uniques 859; 2023-03-19 09:55:20,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:55:20,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:55:20,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219720923, max timestamp=1679219720954Num uniques 1068; 2023-03-19 09:55:20,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-19 09:55:20,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1679219720923, max timestamp=1679219720955Num uniques 1068; 2023-03-19 09:55:20,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48900, new segment data size=7680, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-19 09:55:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7683, totalHeapSize=17712, min timestamp=1679219720923, max timestamp=1679219720955Num uniques 1068; 2023-03-19 09:55:20,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48903, new segment data size=7683, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-19 09:55:20,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:20,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219780982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:20,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219720976, max timestamp=1679219720983Num uniques 1027; 2023-03-19 09:55:20,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219780982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:20,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-19 09:55:20,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219720976, max timestamp=1679219720983Num uniques 1027; 2023-03-19 09:55:20,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-19 09:55:20,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219720976, max timestamp=1679219720983Num uniques 1027; 2023-03-19 09:55:20,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-19 09:55:21,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219781013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219781015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219781016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219781086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219781086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219781122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219781122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219781123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,290 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319f66ec662f6a04e808fd646da50ecaafb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319f66ec662f6a04e808fd646da50ecaafb 2023-03-19 09:55:21,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219781291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,295 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=127, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/35d8092d4c4846439945164aecaf602b 2023-03-19 09:55:21,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219781292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219781330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219781331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219781331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219781596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219781599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46824 deadline: 1679219781635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46876 deadline: 1679219781636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46834 deadline: 1679219781637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:21,704 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/23f3ed0e84d745589e7b64d98d13f26c 2023-03-19 09:55:22,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219782102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219782103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,114 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/79d31b64c91e4e4e9ffe5d61605ffaf5 2023-03-19 09:55:22,117 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/35d8092d4c4846439945164aecaf602b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/35d8092d4c4846439945164aecaf602b 2023-03-19 09:55:22,121 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/35d8092d4c4846439945164aecaf602b, entries=100, sequenceid=127, filesize=18.4 K 2023-03-19 09:55:22,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/23f3ed0e84d745589e7b64d98d13f26c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/23f3ed0e84d745589e7b64d98d13f26c 2023-03-19 09:55:22,125 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/23f3ed0e84d745589e7b64d98d13f26c, entries=100, sequenceid=127, filesize=9.3 K 2023-03-19 09:55:22,125 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/79d31b64c91e4e4e9ffe5d61605ffaf5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/79d31b64c91e4e4e9ffe5d61605ffaf5 2023-03-19 09:55:22,128 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/79d31b64c91e4e4e9ffe5d61605ffaf5, entries=100, sequenceid=127, filesize=9.3 K 2023-03-19 09:55:22,129 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 72808d88f0cb7335353d6abe6e889fc0 in 1266ms, sequenceid=127, compaction requested=false 2023-03-19 09:55:22,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:22,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd732ab[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:22,133 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:22,133 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,134 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:22,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:22,136 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:22,138 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:22,138 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:22,138 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:22,139 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:22,139 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:22,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,143 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:22,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219721012, max timestamp=1679219722142Num uniques 1136; 2023-03-19 09:55:22,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219721012, max timestamp=1679219722142Num uniques 1136; 2023-03-19 09:55:22,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-19 09:55:22,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-19 09:55:22,145 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=69.33 KB heapSize=187.27 KB 2023-03-19 09:55:22,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219721012, max timestamp=1679219722142Num uniques 1136; 2023-03-19 09:55:22,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-19 09:55:22,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:22,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:22,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:22,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:22,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:22,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,148 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:22,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:22,148 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,148 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:22,149 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:22,149 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,148 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,149 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,149 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,149 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,149 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,149 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219722172, max timestamp=1679219722177Num uniques 859; 2023-03-19 09:55:22,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:22,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219722172, max timestamp=1679219722177Num uniques 859; 2023-03-19 09:55:22,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:22,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219722172, max timestamp=1679219722177Num uniques 859; 2023-03-19 09:55:22,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:22,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219722193, max timestamp=1679219722211Num uniques 1018; 2023-03-19 09:55:22,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219722193, max timestamp=1679219722211Num uniques 1018; 2023-03-19 09:55:22,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219722193, max timestamp=1679219722211Num uniques 1018; 2023-03-19 09:55:22,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:55:22,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:55:22,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-19 09:55:22,249 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:22,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:22,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:22,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:22,252 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:22,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:22,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:22,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,254 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:22,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:22,255 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,255 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:22,255 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:22,255 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:22,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:22,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:22,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219722235, max timestamp=1679219722258Num uniques 1127; 2023-03-19 09:55:22,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-19 09:55:22,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219722235, max timestamp=1679219722258Num uniques 1127; 2023-03-19 09:55:22,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-19 09:55:22,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679219722235, max timestamp=1679219722258Num uniques 1127; 2023-03-19 09:55:22,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-19 09:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b1225c9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:22,262 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:22,262 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,264 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,264 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:22,264 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:22,264 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:22,264 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,265 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:22,266 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:22,268 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:22,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:22,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:22,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:22,268 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:22,274 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,274 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,274 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:22,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:22,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,284 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:22,284 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:22,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:22,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219722290, max timestamp=1679219722307Num uniques 1136; 2023-03-19 09:55:22,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219722290, max timestamp=1679219722307Num uniques 1136; 2023-03-19 09:55:22,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-19 09:55:22,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-19 09:55:22,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219722290, max timestamp=1679219722307Num uniques 1136; 2023-03-19 09:55:22,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-19 09:55:22,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679219722336, max timestamp=1679219722353Num uniques 1045; 2023-03-19 09:55:22,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-19 09:55:22,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679219722336, max timestamp=1679219722353Num uniques 1045; 2023-03-19 09:55:22,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-19 09:55:22,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679219722336, max timestamp=1679219722353Num uniques 1045; 2023-03-19 09:55:22,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-19 09:55:22,384 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:22,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:22,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:22,387 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,389 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,389 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:22,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:22,391 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,391 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:22,391 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:22,391 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:22,393 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:22,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510fa92a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:22,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:22,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:22,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:22,402 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219722393, max timestamp=1679219722402Num uniques 1104; 2023-03-19 09:55:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:22,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:22,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-19 09:55:22,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219722393, max timestamp=1679219722402Num uniques 1104; 2023-03-19 09:55:22,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=6870, suffix heap size=142784,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=150 2023-03-19 09:55:22,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219722393, max timestamp=1679219722402Num uniques 1104; 2023-03-19 09:55:22,405 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:22,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:22,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:22,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-19 09:55:22,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:22,408 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:22,409 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,409 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,409 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,411 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:22,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,414 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:22,415 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:22,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679219722431, max timestamp=1679219722444Num uniques 1009; 2023-03-19 09:55:22,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679219722431, max timestamp=1679219722444Num uniques 1013; 2023-03-19 09:55:22,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=9748, suffix heap size=132904,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=213 2023-03-19 09:55:22,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-19 09:55:22,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679219722431, max timestamp=1679219722444Num uniques 1013; 2023-03-19 09:55:22,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-19 09:55:22,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679219722459, max timestamp=1679219722481Num uniques 1072; 2023-03-19 09:55:22,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-19 09:55:22,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679219722459, max timestamp=1679219722481Num uniques 1072; 2023-03-19 09:55:22,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679219722459, max timestamp=1679219722481Num uniques 1072; 2023-03-19 09:55:22,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-19 09:55:22,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-19 09:55:22,515 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:22,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:22,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:22,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:22,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:22,518 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:22,518 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,519 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:22,519 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:22,519 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,519 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,520 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:22,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:22,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:22,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:22,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:22,530 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:22,530 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,533 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b10b0d8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:22,535 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:22,536 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,537 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:22,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:22,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:22,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:22,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:22,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:22,542 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:22,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:22,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:22,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:22,542 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:22,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679219722513, max timestamp=1679219722545Num uniques 1231; 2023-03-19 09:55:22,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=10576, suffix heap size=155992,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=231 2023-03-19 09:55:22,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679219722513, max timestamp=1679219722545Num uniques 1231; 2023-03-19 09:55:22,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679219722513, max timestamp=1679219722545Num uniques 1231; 2023-03-19 09:55:22,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=10576, suffix heap size=155992,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=231 2023-03-19 09:55:22,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56376, new segment data size=10576, suffix heap size=155992,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=231 2023-03-19 09:55:22,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,550 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,555 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:22,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:22,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,555 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:22,555 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:22,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,564 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319bbd0f55fd400419daa9c4e9e4ed64551 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bbd0f55fd400419daa9c4e9e4ed64551 2023-03-19 09:55:22,574 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/1a6499105e2944e6ace173741ba7c903 2023-03-19 09:55:22,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/04250a74ded84801a93a8e09eeacd189 2023-03-19 09:55:22,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/c5b56997b6c64ae28b7bd13136ea6983 2023-03-19 09:55:22,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/1a6499105e2944e6ace173741ba7c903 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/1a6499105e2944e6ace173741ba7c903 2023-03-19 09:55:22,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/1a6499105e2944e6ace173741ba7c903, entries=200, sequenceid=199, filesize=32.1 K 2023-03-19 09:55:22,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/04250a74ded84801a93a8e09eeacd189 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/04250a74ded84801a93a8e09eeacd189 2023-03-19 09:55:22,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679219722581, max timestamp=1679219722614Num uniques 1090; 2023-03-19 09:55:22,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-19 09:55:22,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679219722581, max timestamp=1679219722614Num uniques 1090; 2023-03-19 09:55:22,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679219722581, max timestamp=1679219722614Num uniques 1090; 2023-03-19 09:55:22,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-19 09:55:22,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-19 09:55:22,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/04250a74ded84801a93a8e09eeacd189, entries=150, sequenceid=199, filesize=11.8 K 2023-03-19 09:55:22,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/c5b56997b6c64ae28b7bd13136ea6983 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c5b56997b6c64ae28b7bd13136ea6983 2023-03-19 09:55:22,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c5b56997b6c64ae28b7bd13136ea6983, entries=150, sequenceid=199, filesize=11.8 K 2023-03-19 09:55:22,631 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~63.66 KB/65184, currentSize=60.38 KB/61830 for 72808d88f0cb7335353d6abe6e889fc0 in 486ms, sequenceid=199, compaction requested=true 2023-03-19 09:55:22,631 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:22,631 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:22,631 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:22,631 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:22,631 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:22,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:22,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=67.09 KB heapSize=178.17 KB 2023-03-19 09:55:22,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:22,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:22,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:22,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:22,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:22,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:22,633 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:22,633 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77670 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:22,633 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/B is initiating minor compaction (all files) 2023-03-19 09:55:22,633 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/A is initiating minor compaction (all files) 2023-03-19 09:55:22,633 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/B in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,633 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/A in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,633 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/def4815270f8461f990ca77ca568e311, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/23f3ed0e84d745589e7b64d98d13f26c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/04250a74ded84801a93a8e09eeacd189] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=32.9 K 2023-03-19 09:55:22,634 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0ddd938ec64641c0ad65e1741961f77a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/35d8092d4c4846439945164aecaf602b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/1a6499105e2944e6ace173741ba7c903] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=75.8 K 2023-03-19 09:55:22,634 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting def4815270f8461f990ca77ca568e311, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679219718606 2023-03-19 09:55:22,634 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0ddd938ec64641c0ad65e1741961f77a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679219718606 2023-03-19 09:55:22,634 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 23f3ed0e84d745589e7b64d98d13f26c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679219720786 2023-03-19 09:55:22,635 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 35d8092d4c4846439945164aecaf602b, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679219720786 2023-03-19 09:55:22,635 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 04250a74ded84801a93a8e09eeacd189, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679219721014 2023-03-19 09:55:22,635 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1a6499105e2944e6ace173741ba7c903, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679219721012 2023-03-19 09:55:22,649 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#B#compaction#349 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:22,655 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:22,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:22,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:22,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:22,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:22,660 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:22,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:22,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:22,661 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,661 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,662 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,663 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,663 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:22,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:22,669 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#A#compaction#350 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:22,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:22,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:22,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:22,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:22,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db48f7b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:22,676 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:22,676 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:22,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:22,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:22,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:22,681 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:22,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:22,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:22,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:22,681 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:22,687 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,687 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,688 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/58b937ab762647c6935416bc04d15fa4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/58b937ab762647c6935416bc04d15fa4 2023-03-19 09:55:22,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,694 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/B of 72808d88f0cb7335353d6abe6e889fc0 into 58b937ab762647c6935416bc04d15fa4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:22,694 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:22,694 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/B, priority=13, startTime=1679219722631; duration=0sec 2023-03-19 09:55:22,694 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:22,694 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:22,695 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:22,695 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/C is initiating minor compaction (all files) 2023-03-19 09:55:22,695 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/C in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,695 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/94258e3146594ee39b338f284e00de02, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/79d31b64c91e4e4e9ffe5d61605ffaf5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c5b56997b6c64ae28b7bd13136ea6983] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=32.9 K 2023-03-19 09:55:22,696 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 94258e3146594ee39b338f284e00de02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679219718606 2023-03-19 09:55:22,696 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 79d31b64c91e4e4e9ffe5d61605ffaf5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679219720786 2023-03-19 09:55:22,696 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,696 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,696 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,697 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,696 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,697 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,697 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c5b56997b6c64ae28b7bd13136ea6983, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679219721014 2023-03-19 09:55:22,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:22,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:22,697 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:22,697 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:22,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219782682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,705 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#C#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:22,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219782697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219782698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,717 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/52a0058af9324dd889d0a28255e6f888 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/52a0058af9324dd889d0a28255e6f888 2023-03-19 09:55:22,722 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/C of 72808d88f0cb7335353d6abe6e889fc0 into 52a0058af9324dd889d0a28255e6f888(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:22,722 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:22,722 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/C, priority=13, startTime=1679219722632; duration=0sec 2023-03-19 09:55:22,722 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:22,797 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:22,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:22,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:22,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219782799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:22,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:22,801 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:22,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:22,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:22,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:22,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:22,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:22,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:22,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:22,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,807 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:22,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5782d569[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:22,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:22,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,811 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:22,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:22,813 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:22,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,814 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:22,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:22,816 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:22,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:22,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:22,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:22,816 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:22,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219782813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:22,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219782813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,818 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,818 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,819 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:22,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:22,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,829 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:22,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:22,829 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:22,929 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:22,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:22,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:22,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:22,940 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:22,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:22,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:22,941 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,941 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,942 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:22,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:22,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:22,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:22,950 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,950 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:22,950 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:22,951 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:22,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be8604a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:22,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:22,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,957 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:22,958 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,958 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:22,959 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:22,962 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:22,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:22,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:22,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:22,962 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:22,963 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,963 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,964 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:22,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:22,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,968 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:22,968 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:22,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:22,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:22,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:22,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:22,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:22,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:22,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:22,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:22,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:22,970 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:22,970 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:23,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219783003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219783018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219783018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,066 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319e32314887b4b4d62ba88dde25cc763c1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e32314887b4b4d62ba88dde25cc763c1 2023-03-19 09:55:23,070 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:23,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:23,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:23,073 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=380, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/028385b0402c4cc39b7c386d1305c2ad 2023-03-19 09:55:23,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:23,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:23,074 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:23,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:23,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:23,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:23,075 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,075 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,077 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:23,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:23,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:23,081 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:23,081 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:23,081 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:23,081 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:23,084 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,088 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:23,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:23,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4540899f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:23,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:23,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:23,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:23,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:23,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/77bcda66465f4b1aac7a5691daa5043f 2023-03-19 09:55:23,094 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:23,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:23,105 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:23,105 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:23,106 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:23,106 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:23,106 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:23,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:23,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:23,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219783112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:23,116 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:23,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:23,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:23,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:23,117 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:23,117 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:23,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:23,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:23,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:23,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:23,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219783116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/89ac7ccc6dff42d4a1b05f40899cd500 2023-03-19 09:55:23,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/028385b0402c4cc39b7c386d1305c2ad as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/028385b0402c4cc39b7c386d1305c2ad 2023-03-19 09:55:23,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/028385b0402c4cc39b7c386d1305c2ad, entries=250, sequenceid=380, filesize=39.2 K 2023-03-19 09:55:23,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/77bcda66465f4b1aac7a5691daa5043f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/77bcda66465f4b1aac7a5691daa5043f 2023-03-19 09:55:23,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/77bcda66465f4b1aac7a5691daa5043f, entries=150, sequenceid=380, filesize=11.9 K 2023-03-19 09:55:23,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/89ac7ccc6dff42d4a1b05f40899cd500 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89ac7ccc6dff42d4a1b05f40899cd500 2023-03-19 09:55:23,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89ac7ccc6dff42d4a1b05f40899cd500, entries=150, sequenceid=380, filesize=11.9 K 2023-03-19 09:55:23,157 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~198.05 KB/202800, currentSize=107.34 KB/109920 for 72808d88f0cb7335353d6abe6e889fc0 in 525ms, sequenceid=380, compaction requested=false 2023-03-19 09:55:23,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:23,217 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:23,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:23,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:23,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:23,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:23,221 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:23,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:23,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:23,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:23,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,222 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:23,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:23,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:23,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:23,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:23,224 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:23,224 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:23,224 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-19 09:55:23,225 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:23,225 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:23,225 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:23,225 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:23,225 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:23,225 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:23,246 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031955db6534faef4cd988dcfef93628c610 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031955db6534faef4cd988dcfef93628c610 2023-03-19 09:55:23,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:23,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:23,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219783340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219783341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219783344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219783450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219783450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219783452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,526 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/ccc19a3307cf4f09bc11d5fefcd22f1e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ccc19a3307cf4f09bc11d5fefcd22f1e 2023-03-19 09:55:23,529 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/A of 72808d88f0cb7335353d6abe6e889fc0 into ccc19a3307cf4f09bc11d5fefcd22f1e(size=25.6 K), total size for store is 64.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:23,529 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:23,529 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/A, priority=13, startTime=1679219722631; duration=0sec 2023-03-19 09:55:23,529 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:23,651 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/7aa5165f543341f8a49e2063f8fee5e0 2023-03-19 09:55:23,660 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/0916f8a2db04465ca10ecf5e966ce28d 2023-03-19 09:55:23,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219783658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219783659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219783661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219783964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219783966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:23,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:23,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219783969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,069 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/132fc3a1addb4ba988424333705dabc5 2023-03-19 09:55:24,074 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/7aa5165f543341f8a49e2063f8fee5e0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7aa5165f543341f8a49e2063f8fee5e0 2023-03-19 09:55:24,077 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7aa5165f543341f8a49e2063f8fee5e0, entries=150, sequenceid=401, filesize=25.4 K 2023-03-19 09:55:24,078 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/0916f8a2db04465ca10ecf5e966ce28d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/0916f8a2db04465ca10ecf5e966ce28d 2023-03-19 09:55:24,099 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/0916f8a2db04465ca10ecf5e966ce28d, entries=150, sequenceid=401, filesize=11.9 K 2023-03-19 09:55:24,099 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/132fc3a1addb4ba988424333705dabc5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/132fc3a1addb4ba988424333705dabc5 2023-03-19 09:55:24,102 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/132fc3a1addb4ba988424333705dabc5, entries=150, sequenceid=401, filesize=11.9 K 2023-03-19 09:55:24,103 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 72808d88f0cb7335353d6abe6e889fc0 in 879ms, sequenceid=401, compaction requested=true 2023-03-19 09:55:24,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:24,103 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:24,105 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c76191b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:24,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:24,107 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:24,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:24,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:24,110 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,110 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:24,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:24,112 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:24,112 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:24,112 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:24,112 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:24,112 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:24,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,114 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,115 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,118 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:24,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:24,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,119 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:24,119 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:24,118 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,119 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,119 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,119 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,119 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,219 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:24,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:24,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:24,222 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:24,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:24,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:24,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:24,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:24,226 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:24,226 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:24,243 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319a1e608bcf8974995b6b8e219f206375c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a1e608bcf8974995b6b8e219f206375c 2023-03-19 09:55:24,247 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=415, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/a938530e4e7343d486d01e55480f7226 2023-03-19 09:55:24,269 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/d44ed5a7235f45a48f339f3e50f05b02 2023-03-19 09:55:24,277 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/3171f4d5cea04934bb797e7e6a7cd79c 2023-03-19 09:55:24,281 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/a938530e4e7343d486d01e55480f7226 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a938530e4e7343d486d01e55480f7226 2023-03-19 09:55:24,284 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a938530e4e7343d486d01e55480f7226, entries=150, sequenceid=415, filesize=25.4 K 2023-03-19 09:55:24,285 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/d44ed5a7235f45a48f339f3e50f05b02 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/d44ed5a7235f45a48f339f3e50f05b02 2023-03-19 09:55:24,287 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/d44ed5a7235f45a48f339f3e50f05b02, entries=150, sequenceid=415, filesize=11.9 K 2023-03-19 09:55:24,288 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/3171f4d5cea04934bb797e7e6a7cd79c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/3171f4d5cea04934bb797e7e6a7cd79c 2023-03-19 09:55:24,291 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/3171f4d5cea04934bb797e7e6a7cd79c, entries=150, sequenceid=415, filesize=11.9 K 2023-03-19 09:55:24,292 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 72808d88f0cb7335353d6abe6e889fc0 in 66ms, sequenceid=415, compaction requested=true 2023-03-19 09:55:24,292 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:24,292 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:24,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:24,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1eba61[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:24,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:24,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:24,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:24,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:24,297 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,297 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:24,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:24,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:24,302 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:24,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:24,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:24,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:24,302 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:24,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,304 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,306 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,308 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:24,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:24,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:24,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,309 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:24,309 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:24,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,409 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:24,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:24,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:24,416 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:24,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:24,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:24,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,417 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,418 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:24,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:24,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:24,420 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,420 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:24,420 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:24,420 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,423 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:24,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cc7fb8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:24,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:24,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,426 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:24,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:24,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:24,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:24,431 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:24,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:24,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:24,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:24,431 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:24,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:24,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:24,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,439 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:24,439 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:24,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:24,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:55:24,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:24,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:24,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:24,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:24,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:24,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:24,539 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:24,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:24,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:24,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:24,544 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:24,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:24,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:24,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,545 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,545 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,546 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:24,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:24,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:24,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:24,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:24,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:24,548 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,548 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031933afde41a16f4b8892d0de556bc26cb6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031933afde41a16f4b8892d0de556bc26cb6 2023-03-19 09:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:24,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4811aef[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:24,557 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:24,557 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,559 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:24,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:24,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:24,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:24,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:24,564 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:24,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:24,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:24,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:24,564 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:24,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:24,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:24,569 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:24,570 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:24,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219724572, max timestamp=1679219724581Num uniques 859; 2023-03-19 09:55:24,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219784581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:24,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219724572, max timestamp=1679219724581Num uniques 859; 2023-03-19 09:55:24,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219724572, max timestamp=1679219724581Num uniques 859; 2023-03-19 09:55:24,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:24,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:24,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219784643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219784644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,670 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:24,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:24,675 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:24,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:24,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:24,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:24,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:24,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:24,679 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:24,679 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:24,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:24,680 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:24,681 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83375a5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:24,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:24,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,684 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:24,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:24,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:24,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:24,686 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:24,689 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:24,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:24,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:24,690 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:24,690 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:24,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219784691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:24,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:24,698 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,699 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:24,699 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:24,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219784748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219784752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,799 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:24,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:24,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:24,802 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:24,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:24,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:24,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:24,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:24,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:24,805 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:24,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:24,805 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:24,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:24,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,807 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:24,809 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:24,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f61b70[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:24,809 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,811 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:24,811 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,811 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:24,812 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:24,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:24,814 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:24,814 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:24,814 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:24,814 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:24,814 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:24,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,816 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:24,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:24,821 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,821 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:24,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,821 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:24,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219784896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,921 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:24,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:24,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:24,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:24,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:24,925 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:24,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:24,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:24,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,927 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,927 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:24,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:24,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:24,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:24,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:24,931 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:24,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26618569[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:24,934 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:24,934 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,938 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,938 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:24,940 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:24,944 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:24,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:24,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:24,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:24,944 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:24,946 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,946 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:24,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,946 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:24,947 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,948 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,950 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:24,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:24,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:24,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:24,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:24,950 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:24,950 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:24,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:24,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:24,955 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=426, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/b7721c75c6e34a5c9dc03ea1ca8cfe57 2023-03-19 09:55:24,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219784957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:24,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:24,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219784957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,051 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:25,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:25,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:25,054 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:25,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:25,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:25,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,055 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:25,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:25,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:25,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:25,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:25,057 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,057 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:25,057 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:25,057 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:25,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:25,060 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:25,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2abbb528[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:25,062 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:25,062 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,063 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:25,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:25,063 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:25,063 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,063 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:25,064 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:25,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:25,068 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:25,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:25,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:25,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:25,068 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:25,069 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,069 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,069 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,070 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:25,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:25,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:25,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:25,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:25,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:25,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:25,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:25,074 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,075 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:25,075 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,075 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,075 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:25,075 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:25,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219785125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,133 DEBUG [Thread-2433] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:25,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219785131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,136 DEBUG [Thread-2431] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:25,175 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:25,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:25,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:25,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:25,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:25,178 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:25,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:25,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:25,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,178 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:25,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:25,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:25,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:25,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:25,181 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,181 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:25,181 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:25,181 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:25,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:25,182 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:25,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:25,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39366387[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:25,185 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:25,185 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,186 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:25,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:25,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:25,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:25,188 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:25,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:25,192 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:25,192 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:25,192 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:25,192 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:25,192 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:25,193 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,193 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,193 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:25,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:25,194 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:25,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:25,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:25,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:25,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,197 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:25,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,197 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219785202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219785262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219785262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,297 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:25,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:25,301 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:25,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:25,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:25,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,301 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,303 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:25,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:25,303 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,303 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:25,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:25,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:25,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:25,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:25,305 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,305 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:25,305 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:25,305 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:25,309 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,310 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:25,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92bc24a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:25,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:25,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:25,316 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:25,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:25,318 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:25,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:25,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:25,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:25,318 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:25,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:25,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:25,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:25,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,327 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,327 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:25,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:25,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:25,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:25,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:25,328 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:25,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:25,328 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:25,328 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:25,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:25,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:25,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/acfa223d07d24f93b17499b7e20eb3c4 2023-03-19 09:55:25,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/e30a305cda764bebac61188bd7a65ba7 2023-03-19 09:55:25,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/b7721c75c6e34a5c9dc03ea1ca8cfe57 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/b7721c75c6e34a5c9dc03ea1ca8cfe57 2023-03-19 09:55:25,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/b7721c75c6e34a5c9dc03ea1ca8cfe57, entries=350, sequenceid=426, filesize=52.9 K 2023-03-19 09:55:25,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/acfa223d07d24f93b17499b7e20eb3c4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/acfa223d07d24f93b17499b7e20eb3c4 2023-03-19 09:55:25,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/acfa223d07d24f93b17499b7e20eb3c4, entries=150, sequenceid=426, filesize=11.9 K 2023-03-19 09:55:25,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/e30a305cda764bebac61188bd7a65ba7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e30a305cda764bebac61188bd7a65ba7 2023-03-19 09:55:25,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e30a305cda764bebac61188bd7a65ba7, entries=150, sequenceid=426, filesize=11.9 K 2023-03-19 09:55:25,389 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 72808d88f0cb7335353d6abe6e889fc0 in 854ms, sequenceid=426, compaction requested=true 2023-03-19 09:55:25,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:25,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:25,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:25,389 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:25,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:25,389 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:25,391 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 172674 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:25,391 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61122 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:25,391 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/A is initiating minor compaction (all files) 2023-03-19 09:55:25,391 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/B is initiating minor compaction (all files) 2023-03-19 09:55:25,391 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/A in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,391 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/B in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,391 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ccc19a3307cf4f09bc11d5fefcd22f1e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/028385b0402c4cc39b7c386d1305c2ad, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7aa5165f543341f8a49e2063f8fee5e0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a938530e4e7343d486d01e55480f7226, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/b7721c75c6e34a5c9dc03ea1ca8cfe57] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=168.6 K 2023-03-19 09:55:25,391 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/58b937ab762647c6935416bc04d15fa4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/77bcda66465f4b1aac7a5691daa5043f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/0916f8a2db04465ca10ecf5e966ce28d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/d44ed5a7235f45a48f339f3e50f05b02, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/acfa223d07d24f93b17499b7e20eb3c4] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=59.7 K 2023-03-19 09:55:25,391 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ccc19a3307cf4f09bc11d5fefcd22f1e, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679219721014 2023-03-19 09:55:25,391 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 58b937ab762647c6935416bc04d15fa4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679219721014 2023-03-19 09:55:25,392 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 028385b0402c4cc39b7c386d1305c2ad, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679219722617 2023-03-19 09:55:25,392 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 77bcda66465f4b1aac7a5691daa5043f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679219722622 2023-03-19 09:55:25,392 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7aa5165f543341f8a49e2063f8fee5e0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679219722662 2023-03-19 09:55:25,392 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0916f8a2db04465ca10ecf5e966ce28d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679219722662 2023-03-19 09:55:25,392 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a938530e4e7343d486d01e55480f7226, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219723338 2023-03-19 09:55:25,393 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d44ed5a7235f45a48f339f3e50f05b02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219723338 2023-03-19 09:55:25,393 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b7721c75c6e34a5c9dc03ea1ca8cfe57, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679219724499 2023-03-19 09:55:25,393 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting acfa223d07d24f93b17499b7e20eb3c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679219724526 2023-03-19 09:55:25,402 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#B#compaction#364 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:25,406 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#A#compaction#363 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:25,414 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/67c2cc5fd48d40968583713f3220e4df as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/67c2cc5fd48d40968583713f3220e4df 2023-03-19 09:55:25,419 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/B of 72808d88f0cb7335353d6abe6e889fc0 into 67c2cc5fd48d40968583713f3220e4df(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:25,419 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:25,419 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/B, priority=11, startTime=1679219725389; duration=0sec 2023-03-19 09:55:25,419 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:25,419 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:25,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61122 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:25,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/C is initiating minor compaction (all files) 2023-03-19 09:55:25,423 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/C in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,423 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/52a0058af9324dd889d0a28255e6f888, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89ac7ccc6dff42d4a1b05f40899cd500, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/132fc3a1addb4ba988424333705dabc5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/3171f4d5cea04934bb797e7e6a7cd79c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e30a305cda764bebac61188bd7a65ba7] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=59.7 K 2023-03-19 09:55:25,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 52a0058af9324dd889d0a28255e6f888, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679219721014 2023-03-19 09:55:25,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 89ac7ccc6dff42d4a1b05f40899cd500, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679219722622 2023-03-19 09:55:25,424 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 132fc3a1addb4ba988424333705dabc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679219722662 2023-03-19 09:55:25,424 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 3171f4d5cea04934bb797e7e6a7cd79c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679219723338 2023-03-19 09:55:25,425 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e30a305cda764bebac61188bd7a65ba7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679219724526 2023-03-19 09:55:25,426 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/0d3ebbc11c954386bd77a4656f1c1551 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0d3ebbc11c954386bd77a4656f1c1551 2023-03-19 09:55:25,428 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:25,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:25,432 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/A of 72808d88f0cb7335353d6abe6e889fc0 into 0d3ebbc11c954386bd77a4656f1c1551(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:25,432 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:25,432 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/A, priority=11, startTime=1679219725389; duration=0sec 2023-03-19 09:55:25,432 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:25,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:25,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:25,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:25,435 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:25,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:25,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:25,436 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#C#compaction#365 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:25,436 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,436 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,437 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:25,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:25,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:25,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:25,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:25,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:25,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:25,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:25,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:25,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:25,439 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:25,439 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:25,439 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:55:25,440 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:25,440 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:25,440 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:25,440 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:25,440 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:25,440 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:25,450 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/e4e22edee69f4d3992768ff2d96563de as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e4e22edee69f4d3992768ff2d96563de 2023-03-19 09:55:25,455 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/C of 72808d88f0cb7335353d6abe6e889fc0 into e4e22edee69f4d3992768ff2d96563de(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:25,455 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:25,455 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/C, priority=11, startTime=1679219725389; duration=0sec 2023-03-19 09:55:25,455 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:25,455 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319bd9f584c2c4e4c8588bb4711b29794d7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bd9f584c2c4e4c8588bb4711b29794d7 2023-03-19 09:55:25,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:25,716 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:25,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219785741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219785770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219785772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:25,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219785844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:25,860 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/09bfdcd29ef942e8a2d88f2f349c411b 2023-03-19 09:55:25,872 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/b433e5ba08994e068aadf9b8b358b208 2023-03-19 09:55:26,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219786047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,294 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/89f0008cf3084237963067ed8ac7e6e2 2023-03-19 09:55:26,300 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/09bfdcd29ef942e8a2d88f2f349c411b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/09bfdcd29ef942e8a2d88f2f349c411b 2023-03-19 09:55:26,305 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/09bfdcd29ef942e8a2d88f2f349c411b, entries=150, sequenceid=465, filesize=25.4 K 2023-03-19 09:55:26,306 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/b433e5ba08994e068aadf9b8b358b208 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b433e5ba08994e068aadf9b8b358b208 2023-03-19 09:55:26,310 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b433e5ba08994e068aadf9b8b358b208, entries=150, sequenceid=465, filesize=11.9 K 2023-03-19 09:55:26,311 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/89f0008cf3084237963067ed8ac7e6e2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89f0008cf3084237963067ed8ac7e6e2 2023-03-19 09:55:26,317 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89f0008cf3084237963067ed8ac7e6e2, entries=150, sequenceid=465, filesize=11.9 K 2023-03-19 09:55:26,318 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 72808d88f0cb7335353d6abe6e889fc0 in 879ms, sequenceid=465, compaction requested=false 2023-03-19 09:55:26,318 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:26,318 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:26,320 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:26,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@234b54cb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:26,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:26,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,325 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:26,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:26,325 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:26,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:26,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:26,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:26,329 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:26,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:26,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:26,329 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:26,329 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:26,330 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,331 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,331 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,331 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,332 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,335 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,336 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,336 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,336 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:26,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:26,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:26,336 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:26,336 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:26,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:55:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:26,436 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:26,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:26,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:26,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:26,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:26,439 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:26,439 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:26,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:26,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:26,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:26,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:26,444 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,444 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:26,444 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:26,444 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:26,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219726400, max timestamp=1679219726443Num uniques 859; 2023-03-19 09:55:26,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219726400, max timestamp=1679219726443Num uniques 859; 2023-03-19 09:55:26,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:26,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:26,445 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219726400, max timestamp=1679219726443Num uniques 859; 2023-03-19 09:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,446 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,446 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,446 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:26,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@542ea77f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:26,447 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:26,447 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,448 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:26,449 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,449 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:26,450 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:26,450 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,450 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:26,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:26,452 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:26,452 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:26,452 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:26,457 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,457 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,457 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,460 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:26,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:26,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,461 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:26,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:26,461 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:26,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,561 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:26,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:26,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:26,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:26,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:26,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:26,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219786557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,564 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:26,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:26,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:26,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,565 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:26,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:26,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:26,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:26,570 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:26,573 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:26,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0ecc26[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:26,575 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:26,575 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,577 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,577 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:26,577 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:26,577 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:26,577 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,577 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,578 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:26,580 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:26,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:26,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:26,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:26,580 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:26,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,582 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,589 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:26,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,589 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:26,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:26,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,589 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:26,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:26,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219786665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,689 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:26,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:26,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:26,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:26,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:26,693 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:26,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:26,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:26,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:26,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:26,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:26,696 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,696 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:26,696 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:26,696 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:26,696 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:26,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,698 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,698 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,698 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:26,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e07c3f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:26,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:26,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:26,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:26,704 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:26,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:26,707 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:26,707 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:26,707 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:26,707 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:26,707 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:26,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,711 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:26,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:26,713 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,713 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:26,714 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:26,763 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319828dcc19538e4953b5dc86b1aafea699 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319828dcc19538e4953b5dc86b1aafea699 2023-03-19 09:55:26,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:26,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219786776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:26,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219786784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,814 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:26,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:26,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:26,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:26,817 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:26,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:26,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:26,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,817 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,819 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:26,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:26,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:26,820 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,820 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:26,820 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:26,820 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:26,821 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,821 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:26,821 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:26,823 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,823 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,823 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:26,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f49cc4[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:26,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:26,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,826 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:26,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,827 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:26,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,828 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:26,830 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:26,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:26,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:26,830 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:26,830 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:26,833 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,833 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,833 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,833 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,838 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,838 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:26,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:26,838 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,839 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:26,838 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,839 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:26,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,839 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:26,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:26,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219786870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,939 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:26,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:26,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:26,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:26,943 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:26,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:26,943 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:26,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,945 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,945 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,946 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,946 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:26,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:26,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:26,956 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,956 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:26,956 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:26,956 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:26,958 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:26,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5140c8fa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:26,960 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:26,960 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,961 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:26,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:26,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:26,962 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:26,964 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:26,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:26,967 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:26,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:26,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:26,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:26,967 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:26,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:26,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:26,969 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:26,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:26,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:26,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:26,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:26,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:26,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:26,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:26,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:26,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:26,976 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:26,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:26,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:26,976 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:27,076 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:27,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:27,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:27,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:27,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:27,079 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:27,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:27,079 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:27,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:27,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:27,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:27,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:27,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:27,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,083 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:27,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:27,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:27,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:27,085 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,085 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79875108[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:27,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:27,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,089 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,089 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:27,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:27,089 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:27,090 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:27,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:27,091 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:27,093 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:27,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:27,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:27,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:27,093 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:27,096 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,096 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,096 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:27,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:27,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:27,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:27,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:27,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:27,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:27,101 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:27,101 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:27,168 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=477, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/cee03ba7b8714648b45b555b0154c8b9 2023-03-19 09:55:27,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/c1b7436161dc4f289d0f792647172845 2023-03-19 09:55:27,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:27,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219787177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/49450e8ec2ab48c4b7a03966de46c9b6 2023-03-19 09:55:27,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/cee03ba7b8714648b45b555b0154c8b9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/cee03ba7b8714648b45b555b0154c8b9 2023-03-19 09:55:27,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/cee03ba7b8714648b45b555b0154c8b9, entries=200, sequenceid=477, filesize=32.3 K 2023-03-19 09:55:27,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/c1b7436161dc4f289d0f792647172845 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c1b7436161dc4f289d0f792647172845 2023-03-19 09:55:27,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c1b7436161dc4f289d0f792647172845, entries=150, sequenceid=477, filesize=11.9 K 2023-03-19 09:55:27,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/49450e8ec2ab48c4b7a03966de46c9b6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/49450e8ec2ab48c4b7a03966de46c9b6 2023-03-19 09:55:27,202 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:27,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:27,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/49450e8ec2ab48c4b7a03966de46c9b6, entries=150, sequenceid=477, filesize=11.9 K 2023-03-19 09:55:27,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:27,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 72808d88f0cb7335353d6abe6e889fc0 in 852ms, sequenceid=477, compaction requested=true 2023-03-19 09:55:27,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:27,204 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:27,204 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:27,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:27,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:27,204 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:27,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:27,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:27,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,205 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,206 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:27,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:27,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:27,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:27,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:27,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:27,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:27,208 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:27,208 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:27,208 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/A is initiating minor compaction (all files) 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:27,208 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/A in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,208 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:27,208 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:27,208 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0d3ebbc11c954386bd77a4656f1c1551, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/09bfdcd29ef942e8a2d88f2f349c411b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/cee03ba7b8714648b45b555b0154c8b9] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=83.7 K 2023-03-19 09:55:27,208 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/B is initiating minor compaction (all files) 2023-03-19 09:55:27,209 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/B in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,209 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/67c2cc5fd48d40968583713f3220e4df, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b433e5ba08994e068aadf9b8b358b208, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c1b7436161dc4f289d0f792647172845] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=36.2 K 2023-03-19 09:55:27,209 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0d3ebbc11c954386bd77a4656f1c1551, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679219724526 2023-03-19 09:55:27,209 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 67c2cc5fd48d40968583713f3220e4df, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679219724526 2023-03-19 09:55:27,209 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 09bfdcd29ef942e8a2d88f2f349c411b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219724615 2023-03-19 09:55:27,209 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b433e5ba08994e068aadf9b8b358b208, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219724615 2023-03-19 09:55:27,210 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting cee03ba7b8714648b45b555b0154c8b9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219725722 2023-03-19 09:55:27,212 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c1b7436161dc4f289d0f792647172845, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219725722 2023-03-19 09:55:27,229 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#B#compaction#373 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:27,230 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#A#compaction#372 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:27,254 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/9cbf1caa573040b2a902073251c47296 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/9cbf1caa573040b2a902073251c47296 2023-03-19 09:55:27,260 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/B of 72808d88f0cb7335353d6abe6e889fc0 into 9cbf1caa573040b2a902073251c47296(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:27,260 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:27,260 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/B, priority=13, startTime=1679219727204; duration=0sec 2023-03-19 09:55:27,260 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:27,260 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:27,261 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:27,261 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/C is initiating minor compaction (all files) 2023-03-19 09:55:27,261 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/C in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,261 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e4e22edee69f4d3992768ff2d96563de, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89f0008cf3084237963067ed8ac7e6e2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/49450e8ec2ab48c4b7a03966de46c9b6] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=36.2 K 2023-03-19 09:55:27,262 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e4e22edee69f4d3992768ff2d96563de, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679219724526 2023-03-19 09:55:27,262 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 89f0008cf3084237963067ed8ac7e6e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679219724615 2023-03-19 09:55:27,262 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 49450e8ec2ab48c4b7a03966de46c9b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219725722 2023-03-19 09:55:27,273 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#C#compaction#375 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:27,685 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303198bcf2a0da469414390800aaba082a07e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198bcf2a0da469414390800aaba082a07e 2023-03-19 09:55:27,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:27,688 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:27,690 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=513, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/a92416197b9a4c22a3f2dbb4881883ab 2023-03-19 09:55:27,694 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/4db88d4dbd604eafb5633fcef0dd6a29 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4db88d4dbd604eafb5633fcef0dd6a29 2023-03-19 09:55:27,694 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/bb30057c3ef441c49094ff7af9630fa5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/bb30057c3ef441c49094ff7af9630fa5 2023-03-19 09:55:27,701 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/C of 72808d88f0cb7335353d6abe6e889fc0 into 4db88d4dbd604eafb5633fcef0dd6a29(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:27,701 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/A of 72808d88f0cb7335353d6abe6e889fc0 into bb30057c3ef441c49094ff7af9630fa5(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:27,701 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:27,701 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/A, priority=13, startTime=1679219727204; duration=0sec 2023-03-19 09:55:27,701 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:27,701 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/C, priority=13, startTime=1679219727204; duration=0sec 2023-03-19 09:55:27,701 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:27,701 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:27,702 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/839dcc6ef13a422fa08ff79a4cb36552 2023-03-19 09:55:27,721 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/43d5acc597c344ceb611e6f4e79a5c11 2023-03-19 09:55:27,725 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/a92416197b9a4c22a3f2dbb4881883ab as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a92416197b9a4c22a3f2dbb4881883ab 2023-03-19 09:55:27,728 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a92416197b9a4c22a3f2dbb4881883ab, entries=150, sequenceid=513, filesize=25.4 K 2023-03-19 09:55:27,729 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/839dcc6ef13a422fa08ff79a4cb36552 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/839dcc6ef13a422fa08ff79a4cb36552 2023-03-19 09:55:27,733 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/839dcc6ef13a422fa08ff79a4cb36552, entries=150, sequenceid=513, filesize=11.9 K 2023-03-19 09:55:27,734 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/43d5acc597c344ceb611e6f4e79a5c11 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/43d5acc597c344ceb611e6f4e79a5c11 2023-03-19 09:55:27,740 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/43d5acc597c344ceb611e6f4e79a5c11, entries=150, sequenceid=513, filesize=11.9 K 2023-03-19 09:55:27,741 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=46.96 KB/48090 for 72808d88f0cb7335353d6abe6e889fc0 in 533ms, sequenceid=513, compaction requested=false 2023-03-19 09:55:27,741 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:27,741 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:27,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:27,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:55:27,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:27,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:27,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:27,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:27,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:27,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738ccb83[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:27,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:27,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,752 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:27,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:27,753 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,753 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:27,755 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:27,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,755 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,755 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:27,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:27,758 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:27,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:27,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:27,758 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:27,758 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:27,759 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,759 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,759 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:27,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,762 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:27,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:27,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:27,766 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:27,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:27,766 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:27,766 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:27,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:27,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219727813, max timestamp=1679219727860Num uniques 859; 2023-03-19 09:55:27,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:27,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219727813, max timestamp=1679219727860Num uniques 859; 2023-03-19 09:55:27,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219727813, max timestamp=1679219727860Num uniques 859; 2023-03-19 09:55:27,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:27,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:27,866 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:27,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:27,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:27,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:27,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:27,869 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:27,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:27,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:27,870 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,870 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,872 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:27,872 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:27,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:27,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:27,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:27,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:27,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:27,874 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,874 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:27,874 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:27,874 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:27,874 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:27,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:27,877 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:27,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ca7b8a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:27,877 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:27,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:27,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:27,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,880 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:27,882 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:27,882 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:27,882 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:27,882 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:27,882 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:27,886 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,886 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:27,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,886 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:27,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,888 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:27,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:27,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:27,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:27,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:27,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:27,891 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:27,891 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:27,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219787950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,991 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:27,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:27,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:27,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:27,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:27,994 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:27,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:27,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:27,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,996 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:27,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:27,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:27,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:27,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:27,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:27,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:27,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:27,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:27,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:27,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:27,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:27,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:27,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:27,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:27,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,003 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,003 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,004 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,004 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@709a7091[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,006 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,006 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,006 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,006 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,006 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,010 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,011 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,013 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,013 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,013 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,014 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,015 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,017 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,017 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:28,017 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:28,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219788058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,118 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:28,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:28,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:28,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:28,120 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:28,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:28,120 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:28,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,123 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,123 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:28,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:28,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:28,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:28,123 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:28,123 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,123 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:28,124 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,124 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:28,124 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:28,124 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,125 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,126 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,126 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3356709b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,129 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,131 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,131 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,132 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,132 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,132 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,132 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,133 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,135 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,137 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,137 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,137 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,138 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,138 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,174 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319ac0b21d521d9435eb7394644de16980f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ac0b21d521d9435eb7394644de16980f 2023-03-19 09:55:28,180 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=527, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/dbe95b71aea74e9b9829eb7e5ef8e647 2023-03-19 09:55:28,238 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:28,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:28,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:28,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:28,241 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:28,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:28,241 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:28,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,244 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,244 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:28,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:28,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:28,245 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,245 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:28,245 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:28,246 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:28,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:28,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:28,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,247 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@261375a1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,249 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,250 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,252 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:28,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,254 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,254 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,254 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,257 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,258 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:28,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,261 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,261 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,260 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,261 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:28,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219788265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,361 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:28,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:28,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:28,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:28,364 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:28,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:28,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:28,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:28,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:28,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:28,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:28,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:28,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:28,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:28,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:28,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:28,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,370 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f15cc2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,371 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,371 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,376 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,376 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,376 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:28,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,379 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,380 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,380 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,381 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,383 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,385 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,386 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,386 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,486 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:28,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:28,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:28,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:28,489 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:28,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:28,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:28,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:28,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:28,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:28,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:28,493 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,493 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:28,493 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:28,493 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,494 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,495 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ccb0f5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,496 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,496 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,497 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:28,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,502 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,502 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,506 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,508 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:28,509 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:28,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219788571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/aa051808a7e944a5be162518d63d3aba 2023-03-19 09:55:28,609 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:28,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:28,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:28,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:28,612 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:28,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:28,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:28,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:28,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:28,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:28,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:28,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:28,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:28,616 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,618 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@719ec4ff[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,622 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,623 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,623 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,623 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,623 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:28,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,626 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,626 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,628 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,629 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,631 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,632 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:28,632 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,632 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,632 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,632 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,632 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,733 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:28,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:28,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:28,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:28,737 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:28,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:28,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:28,738 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,738 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,740 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,740 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:28,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:28,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:28,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:28,743 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,743 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:28,743 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:28,743 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1d1910[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,757 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,757 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:28,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,761 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,762 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,762 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,762 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,762 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,764 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,764 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,764 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,769 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,772 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,772 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,774 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,775 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,774 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,775 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,775 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,775 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,775 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:28,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219788784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,792 DEBUG [Thread-2437] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:28,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:28,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219788801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,807 DEBUG [Thread-2435] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:28,875 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:28,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:28,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:28,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:28,878 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:28,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:28,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:28,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:28,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:28,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:28,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:28,882 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,882 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:28,882 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:28,882 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:28,883 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,884 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,884 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,884 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d25b1d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:28,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:28,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,887 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:28,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:28,888 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:28,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:28,891 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:28,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:28,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:28,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:28,891 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:28,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,894 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:28,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:28,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:28,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:28,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:28,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:28,898 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:28,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:28,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:28,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:28,898 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:28,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,899 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:28,899 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:28,899 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:28,999 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:28,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/324adb2b20f84b50b30920098f1f8d8a 2023-03-19 09:55:29,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,002 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,002 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/dbe95b71aea74e9b9829eb7e5ef8e647 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/dbe95b71aea74e9b9829eb7e5ef8e647 2023-03-19 09:55:29,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,015 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,015 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,015 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/dbe95b71aea74e9b9829eb7e5ef8e647, entries=200, sequenceid=527, filesize=32.3 K 2023-03-19 09:55:29,018 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/aa051808a7e944a5be162518d63d3aba as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/aa051808a7e944a5be162518d63d3aba 2023-03-19 09:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d0a6a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,020 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,020 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,023 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,023 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,024 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/aa051808a7e944a5be162518d63d3aba, entries=150, sequenceid=527, filesize=11.9 K 2023-03-19 09:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/324adb2b20f84b50b30920098f1f8d8a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/324adb2b20f84b50b30920098f1f8d8a 2023-03-19 09:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,028 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,028 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/324adb2b20f84b50b30920098f1f8d8a, entries=150, sequenceid=527, filesize=11.9 K 2023-03-19 09:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 72808d88f0cb7335353d6abe6e889fc0 in 1287ms, sequenceid=527, compaction requested=true 2023-03-19 09:55:29,032 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,032 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:29,032 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:29,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:29,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:29,032 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,033 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:29,033 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:29,033 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/A is initiating minor compaction (all files) 2023-03-19 09:55:29,033 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/B is initiating minor compaction (all files) 2023-03-19 09:55:29,033 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/B in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,033 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/A in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,033 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/bb30057c3ef441c49094ff7af9630fa5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a92416197b9a4c22a3f2dbb4881883ab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/dbe95b71aea74e9b9829eb7e5ef8e647] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=83.8 K 2023-03-19 09:55:29,033 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/9cbf1caa573040b2a902073251c47296, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/839dcc6ef13a422fa08ff79a4cb36552, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/aa051808a7e944a5be162518d63d3aba] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=36.3 K 2023-03-19 09:55:29,034 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting bb30057c3ef441c49094ff7af9630fa5, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219725722 2023-03-19 09:55:29,034 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9cbf1caa573040b2a902073251c47296, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219725722 2023-03-19 09:55:29,034 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a92416197b9a4c22a3f2dbb4881883ab, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1679219726523 2023-03-19 09:55:29,034 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 839dcc6ef13a422fa08ff79a4cb36552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1679219726523 2023-03-19 09:55:29,035 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting dbe95b71aea74e9b9829eb7e5ef8e647, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219727708 2023-03-19 09:55:29,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,035 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting aa051808a7e944a5be162518d63d3aba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219727708 2023-03-19 09:55:29,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,036 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,036 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,041 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#B#compaction#382 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:29,045 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#A#compaction#381 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:29,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:29,084 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:55:29,084 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:29,084 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:29,084 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:29,084 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:29,084 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:29,084 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:29,136 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:29,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,138 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,139 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,139 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,139 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,139 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,139 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:29,142 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,142 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,142 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219789131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,143 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,143 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,144 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,144 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7338fb8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,147 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,147 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,147 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,148 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:29,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1679219789141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,152 DEBUG [Thread-2431] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,153 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,153 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:29,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46880 deadline: 1679219789149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,157 DEBUG [Thread-2433] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:29,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:29,164 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,164 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,164 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,165 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219789244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,264 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,268 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,268 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,269 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,273 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c2c3d7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,278 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,279 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,284 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,288 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,288 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,288 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,288 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,288 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,288 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,288 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,288 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,388 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:29,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,391 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,394 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,394 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,394 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,395 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,396 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,396 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,396 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,396 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,397 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,398 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,398 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254e7e01[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,399 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,399 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,400 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,401 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,401 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,402 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,404 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,404 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,406 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,410 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,410 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:29,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219789449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,466 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/49078cadf14e4a4f9f9baa26a28b3eed as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49078cadf14e4a4f9f9baa26a28b3eed 2023-03-19 09:55:29,472 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/B of 72808d88f0cb7335353d6abe6e889fc0 into 49078cadf14e4a4f9f9baa26a28b3eed(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:29,472 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:29,472 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/B, priority=13, startTime=1679219729032; duration=0sec 2023-03-19 09:55:29,472 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:29,472 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:29,473 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:29,473 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/C is initiating minor compaction (all files) 2023-03-19 09:55:29,473 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/C in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,473 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4db88d4dbd604eafb5633fcef0dd6a29, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/43d5acc597c344ceb611e6f4e79a5c11, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/324adb2b20f84b50b30920098f1f8d8a] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=36.3 K 2023-03-19 09:55:29,474 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4db88d4dbd604eafb5633fcef0dd6a29, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679219725722 2023-03-19 09:55:29,474 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 43d5acc597c344ceb611e6f4e79a5c11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1679219726523 2023-03-19 09:55:29,474 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 324adb2b20f84b50b30920098f1f8d8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219727708 2023-03-19 09:55:29,475 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/5884fff876594dd590b54588e3811df8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/5884fff876594dd590b54588e3811df8 2023-03-19 09:55:29,481 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/A of 72808d88f0cb7335353d6abe6e889fc0 into 5884fff876594dd590b54588e3811df8(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:29,481 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:29,481 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/A, priority=13, startTime=1679219729032; duration=0sec 2023-03-19 09:55:29,481 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:29,483 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#C#compaction#384 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:29,496 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303195aee43a7e3c9430ca2dd4146a591c070 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195aee43a7e3c9430ca2dd4146a591c070 2023-03-19 09:55:29,510 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:29,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,513 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,513 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,514 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,514 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,515 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,515 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,516 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,516 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,518 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,518 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,518 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,518 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,520 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,522 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df8ee98[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,524 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,526 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,526 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,526 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,526 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,528 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,528 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,530 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,531 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,538 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,539 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,538 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,539 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,539 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,639 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,642 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,642 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,642 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,642 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,642 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,642 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,643 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@562df6d0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,651 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,651 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,655 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,655 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,655 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,655 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,655 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,662 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,662 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,662 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,662 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:29,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219789755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,762 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:29,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,765 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,765 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,768 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,768 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,770 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,771 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,771 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,772 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,772 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63c45bb9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,773 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,773 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,774 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,774 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,776 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,778 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,778 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,782 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,784 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,785 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,785 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,885 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:29,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:29,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:29,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:29,888 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:29,888 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,888 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:29,888 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:29,889 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,889 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:29,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:29,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:29,891 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:29,891 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:29,891 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:29,891 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:29,892 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,892 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:29,892 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:29,892 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:29,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:29,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,894 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,894 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,894 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:29,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ac9fb6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:29,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:29,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,897 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/6de192fdc8bd422da2850821761cb885 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6de192fdc8bd422da2850821761cb885 2023-03-19 09:55:29,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:29,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:29,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:29,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:29,900 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/C of 72808d88f0cb7335353d6abe6e889fc0 into 6de192fdc8bd422da2850821761cb885(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:29,900 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:29,900 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/C, priority=13, startTime=1679219729032; duration=0sec 2023-03-19 09:55:29,900 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,901 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:29,903 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:29,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:29,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:29,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:29,903 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:29,904 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:29,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:29,905 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=564, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/2f269d372a97424ca9efee6c69b11107 2023-03-19 09:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:29,906 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,907 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:29,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:29,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:29,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:29,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:29,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:29,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:29,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:29,910 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:29,910 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:29,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:29,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:29,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,010 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:30,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:30,013 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:30,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:30,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:30,014 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,014 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,019 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,019 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,020 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,020 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:30,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:30,022 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,022 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:30,022 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:30,022 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:30,028 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,029 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,029 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:30,029 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f2a2bb5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:30,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:30,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:30,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:30,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:30,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:30,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:30,040 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:30,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:30,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:30,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:30,040 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:30,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,044 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,045 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,047 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,047 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:30,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,047 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,048 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:30,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,048 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:30,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:30,148 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:30,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:30,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:30,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:30,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:30,151 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:30,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:30,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:30,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,153 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,153 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:30,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:30,154 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,154 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:30,155 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:30,155 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:30,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:30,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:30,155 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:30,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,157 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,157 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:30,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688d1f6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:30,159 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:30,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,163 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:30,166 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:30,166 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:30,166 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:30,166 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:30,166 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:30,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,168 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,171 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:30,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:30,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:30,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,174 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,174 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:30,174 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:30,174 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:30,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219790262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,274 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:30,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:30,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:30,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:30,277 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:30,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:30,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:30,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,279 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,279 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,279 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,280 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:30,282 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,282 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:30,282 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:30,282 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:30,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47d04b0d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:30,286 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:30,286 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:30,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:30,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:30,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:30,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:30,291 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:30,292 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:30,292 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:30,292 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:30,292 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:30,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,294 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,296 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:30,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:30,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,299 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,299 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,299 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:30,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/72e7f29815d54edbaa80d4846b22f464 2023-03-19 09:55:30,399 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:30,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:30,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:30,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:30,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:30,402 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:30,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:30,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:30,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,403 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:30,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:30,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:30,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:30,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:30,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:30,405 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:30,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:30,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:30,406 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:30,407 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,407 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,408 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:30,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:30,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16502d6e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:30,414 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:30,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:30,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:30,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:30,415 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:30,418 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:30,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:30,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:30,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:30,418 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:30,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,420 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,423 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:30,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:30,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,425 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:30,425 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,425 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,525 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:30,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:30,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:30,528 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:30,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:30,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:30,530 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,530 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,533 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:30,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:30,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:30,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:30,538 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,538 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:30,538 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:30,538 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:30,538 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:30,540 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,540 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c9db92[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:30,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:30,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,543 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,543 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:30,543 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,543 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:30,544 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:30,544 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:30,545 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:30,547 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:30,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:30,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:30,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:30,547 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:30,549 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,549 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,550 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,552 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:30,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:30,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,555 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,555 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:30,555 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:30,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,655 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:30,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:30,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:30,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:30,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:30,658 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:30,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:30,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:30,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,660 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:30,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:30,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:30,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:30,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:30,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:30,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:30,661 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:30,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,661 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:30,662 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:30,662 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,662 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:30,662 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:30,662 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,662 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:30,662 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:30,663 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,663 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@380ea21c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:30,665 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:30,665 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:30,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:30,670 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,670 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:30,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:30,672 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:30,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:30,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:30,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:30,672 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:30,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:30,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,674 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:30,674 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,675 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,677 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,677 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,677 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:30,677 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:30,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:30,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:30,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,678 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:30,678 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:30,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:30,678 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:30,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:30,724 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/2882637e009b4063919264423fd9bb6e 2023-03-19 09:55:30,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/2f269d372a97424ca9efee6c69b11107 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f269d372a97424ca9efee6c69b11107 2023-03-19 09:55:30,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f269d372a97424ca9efee6c69b11107, entries=200, sequenceid=564, filesize=32.3 K 2023-03-19 09:55:30,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/72e7f29815d54edbaa80d4846b22f464 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/72e7f29815d54edbaa80d4846b22f464 2023-03-19 09:55:30,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/72e7f29815d54edbaa80d4846b22f464, entries=150, sequenceid=564, filesize=11.9 K 2023-03-19 09:55:30,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/2882637e009b4063919264423fd9bb6e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/2882637e009b4063919264423fd9bb6e 2023-03-19 09:55:30,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/2882637e009b4063919264423fd9bb6e, entries=150, sequenceid=564, filesize=11.9 K 2023-03-19 09:55:30,742 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 72808d88f0cb7335353d6abe6e889fc0 in 1657ms, sequenceid=564, compaction requested=false 2023-03-19 09:55:30,742 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:30,778 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:30,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:30,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:30,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:30,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:30,781 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:30,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:30,781 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:30,782 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,782 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,784 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:30,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:30,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:30,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:30,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:30,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:30,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:30,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:30,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:30,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:30,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:30,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:30,786 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:30,786 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:30,786 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:55:30,786 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:30,787 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:30,787 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:30,787 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:30,787 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:30,787 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:31,200 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319a748fe262c0b4e30ba652a0c67aaf095 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a748fe262c0b4e30ba652a0c67aaf095 2023-03-19 09:55:31,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:31,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:31,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219731327, max timestamp=1679219731353Num uniques 859; 2023-03-19 09:55:31,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219731327, max timestamp=1679219731353Num uniques 859; 2023-03-19 09:55:31,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219731327, max timestamp=1679219731353Num uniques 859; 2023-03-19 09:55:31,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:31,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:31,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:31,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:31,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219791443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:31,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:31,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219791553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:31,605 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=577, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/79b4830af76a4ff3a597a7273d28e1d1 2023-03-19 09:55:31,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:31,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219791761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,015 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/296214c92c9746779c205bba6ca9da2d 2023-03-19 09:55:32,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219792067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,424 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/005fd474cd1d47cebaacc48c3faa01dc 2023-03-19 09:55:32,427 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/79b4830af76a4ff3a597a7273d28e1d1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/79b4830af76a4ff3a597a7273d28e1d1 2023-03-19 09:55:32,430 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/79b4830af76a4ff3a597a7273d28e1d1, entries=150, sequenceid=577, filesize=25.4 K 2023-03-19 09:55:32,431 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/296214c92c9746779c205bba6ca9da2d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/296214c92c9746779c205bba6ca9da2d 2023-03-19 09:55:32,434 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/296214c92c9746779c205bba6ca9da2d, entries=150, sequenceid=577, filesize=11.9 K 2023-03-19 09:55:32,434 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/005fd474cd1d47cebaacc48c3faa01dc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/005fd474cd1d47cebaacc48c3faa01dc 2023-03-19 09:55:32,437 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/005fd474cd1d47cebaacc48c3faa01dc, entries=150, sequenceid=577, filesize=11.9 K 2023-03-19 09:55:32,438 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 72808d88f0cb7335353d6abe6e889fc0 in 1652ms, sequenceid=577, compaction requested=true 2023-03-19 09:55:32,438 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:32,438 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:32,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:32,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:32,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:32,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:32,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:32,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:32,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc30e33[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:32,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:32,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:32,443 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,443 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:32,447 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:32,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:32,450 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:32,450 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:32,450 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:32,450 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:32,450 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:32,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:32,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:32,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:32,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:32,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:32,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:32,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:32,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:32,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:32,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:32,460 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:32,460 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:32,560 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:32,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:32,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:32,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:32,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:32,563 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:32,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:32,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:32,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:32,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:32,566 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:32,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:32,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,566 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:32,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:32,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:32,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:32,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:32,567 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:32,567 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:32,567 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:32,567 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:32,567 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:32,567 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:32,567 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. started... 2023-03-19 09:55:32,568 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:55:32,568 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:32,568 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:32,568 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:32,568 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:32,568 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:32,568 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:32,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:32,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. as already flushing 2023-03-19 09:55:32,589 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319b4a00ffabc8d453c9e022b5430ec6d0f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b4a00ffabc8d453c9e022b5430ec6d0f 2023-03-19 09:55:32,607 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=614, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/2f0de3be3f884366820ed375703a6e30 2023-03-19 09:55:32,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:32,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219792619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:32,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219792730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46876 deadline: 1679219792827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,834 DEBUG [Thread-2437] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:32,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:32,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46824 deadline: 1679219792840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:32,846 DEBUG [Thread-2435] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:32,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:32,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219792936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,034 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/e4cc5d3b132a4af59854adaef0e8edab 2023-03-19 09:55:33,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:33,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46834 deadline: 1679219793242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,445 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/b26b53c6fc7f4570a25f09daca91706f 2023-03-19 09:55:33,449 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/2f0de3be3f884366820ed375703a6e30 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f0de3be3f884366820ed375703a6e30 2023-03-19 09:55:33,453 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f0de3be3f884366820ed375703a6e30, entries=150, sequenceid=614, filesize=25.4 K 2023-03-19 09:55:33,453 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/e4cc5d3b132a4af59854adaef0e8edab as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e4cc5d3b132a4af59854adaef0e8edab 2023-03-19 09:55:33,457 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e4cc5d3b132a4af59854adaef0e8edab, entries=150, sequenceid=614, filesize=11.9 K 2023-03-19 09:55:33,458 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/b26b53c6fc7f4570a25f09daca91706f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/b26b53c6fc7f4570a25f09daca91706f 2023-03-19 09:55:33,462 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/b26b53c6fc7f4570a25f09daca91706f, entries=150, sequenceid=614, filesize=11.9 K 2023-03-19 09:55:33,462 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 72808d88f0cb7335353d6abe6e889fc0 in 895ms, sequenceid=614, compaction requested=true 2023-03-19 09:55:33,462 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:33,462 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:33,462 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:33,462 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:33,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:33,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:33,464 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,464 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:33,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:33,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6f67b8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:33,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:33,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,467 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:33,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:33,469 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:33,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:33,471 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:33,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:33,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:33,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:33,472 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:33,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:33,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:33,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:33,474 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:33,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:33,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,477 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:33,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:33,479 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:33,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:33,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:33,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:33,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:33,479 DEBUG [Thread-2439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:33,480 DEBUG [Thread-2439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:33,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:33,479 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:33,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:33,480 DEBUG [Thread-2440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:54245 2023-03-19 09:55:33,480 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,480 DEBUG [Thread-2440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:33,480 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:33,480 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:33,482 DEBUG [Thread-2444] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:54245 2023-03-19 09:55:33,482 DEBUG [Thread-2444] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:33,490 DEBUG [Thread-2446] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:54245 2023-03-19 09:55:33,490 DEBUG [Thread-2446] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:33,501 DEBUG [Thread-2442] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:54245 2023-03-19 09:55:33,501 DEBUG [Thread-2442] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:33,502 DEBUG [Thread-2448] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eb8f142 to 127.0.0.1:54245 2023-03-19 09:55:33,502 DEBUG [Thread-2448] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:33,580 DEBUG [Thread-2439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:33,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:33,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:33,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:55:33,749 DEBUG [Thread-2429] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:54245 2023-03-19 09:55:33,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:33,749 DEBUG [Thread-2429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:33,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:33,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:33,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:33,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:33,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:34,166 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319d42ae1f7fee14d288533ea717342befd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319d42ae1f7fee14d288533ea717342befd 2023-03-19 09:55:34,571 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=625, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/0358470fe776458d93724693dafd55cc 2023-03-19 09:55:34,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/cee6390a8aa64642bf0071ac8c54a966 2023-03-19 09:55:35,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/fd696cdb187e4cd99dad67efacc902b9 2023-03-19 09:55:35,390 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/0358470fe776458d93724693dafd55cc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0358470fe776458d93724693dafd55cc 2023-03-19 09:55:35,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0358470fe776458d93724693dafd55cc, entries=100, sequenceid=625, filesize=18.6 K 2023-03-19 09:55:35,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/cee6390a8aa64642bf0071ac8c54a966 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/cee6390a8aa64642bf0071ac8c54a966 2023-03-19 09:55:35,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/cee6390a8aa64642bf0071ac8c54a966, entries=100, sequenceid=625, filesize=9.5 K 2023-03-19 09:55:35,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/fd696cdb187e4cd99dad67efacc902b9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/fd696cdb187e4cd99dad67efacc902b9 2023-03-19 09:55:35,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/fd696cdb187e4cd99dad67efacc902b9, entries=100, sequenceid=625, filesize=9.5 K 2023-03-19 09:55:35,399 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 72808d88f0cb7335353d6abe6e889fc0 in 1650ms, sequenceid=625, compaction requested=true 2023-03-19 09:55:35,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:35,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:35,399 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:35,399 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:35,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:35,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:35,400 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130938 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:35,400 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/A is initiating minor compaction (all files) 2023-03-19 09:55:35,400 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:35,400 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/A in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:35,400 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/B is initiating minor compaction (all files) 2023-03-19 09:55:35,400 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/5884fff876594dd590b54588e3811df8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f269d372a97424ca9efee6c69b11107, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/79b4830af76a4ff3a597a7273d28e1d1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f0de3be3f884366820ed375703a6e30, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0358470fe776458d93724693dafd55cc] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=127.9 K 2023-03-19 09:55:35,400 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/B in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:35,401 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49078cadf14e4a4f9f9baa26a28b3eed, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/72e7f29815d54edbaa80d4846b22f464, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/296214c92c9746779c205bba6ca9da2d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e4cc5d3b132a4af59854adaef0e8edab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/cee6390a8aa64642bf0071ac8c54a966] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=57.8 K 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5884fff876594dd590b54588e3811df8, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219727708 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 49078cadf14e4a4f9f9baa26a28b3eed, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219727708 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2f269d372a97424ca9efee6c69b11107, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679219727929 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 72e7f29815d54edbaa80d4846b22f464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679219727929 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 79b4830af76a4ff3a597a7273d28e1d1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679219729120 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 296214c92c9746779c205bba6ca9da2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679219729120 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2f0de3be3f884366820ed375703a6e30, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1679219731404 2023-03-19 09:55:35,401 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e4cc5d3b132a4af59854adaef0e8edab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1679219731404 2023-03-19 09:55:35,402 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0358470fe776458d93724693dafd55cc, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1679219732589 2023-03-19 09:55:35,402 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting cee6390a8aa64642bf0071ac8c54a966, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1679219732589 2023-03-19 09:55:35,408 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#B#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:35,410 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#A#compaction#396 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:35,816 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/065290ce7ec848748d4652eacd9373ed as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/065290ce7ec848748d4652eacd9373ed 2023-03-19 09:55:35,819 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/B of 72808d88f0cb7335353d6abe6e889fc0 into 065290ce7ec848748d4652eacd9373ed(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:35,819 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:35,820 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/B, priority=11, startTime=1679219735399; duration=0sec 2023-03-19 09:55:35,820 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:35,820 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:55:35,821 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:55:35,821 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): 72808d88f0cb7335353d6abe6e889fc0/C is initiating minor compaction (all files) 2023-03-19 09:55:35,821 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72808d88f0cb7335353d6abe6e889fc0/C in TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:35,821 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6de192fdc8bd422da2850821761cb885, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/2882637e009b4063919264423fd9bb6e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/005fd474cd1d47cebaacc48c3faa01dc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/b26b53c6fc7f4570a25f09daca91706f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/fd696cdb187e4cd99dad67efacc902b9] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp, totalSize=57.8 K 2023-03-19 09:55:35,821 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6de192fdc8bd422da2850821761cb885, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679219727708 2023-03-19 09:55:35,821 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2882637e009b4063919264423fd9bb6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679219727929 2023-03-19 09:55:35,822 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 005fd474cd1d47cebaacc48c3faa01dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679219729120 2023-03-19 09:55:35,822 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b26b53c6fc7f4570a25f09daca91706f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1679219731404 2023-03-19 09:55:35,822 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fd696cdb187e4cd99dad67efacc902b9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1679219732589 2023-03-19 09:55:35,828 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): 72808d88f0cb7335353d6abe6e889fc0#C#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:36,235 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/7724f0ad48e04dc9978f945a92baa3c8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/7724f0ad48e04dc9978f945a92baa3c8 2023-03-19 09:55:36,238 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/C of 72808d88f0cb7335353d6abe6e889fc0 into 7724f0ad48e04dc9978f945a92baa3c8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:36,238 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:36,238 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/C, priority=11, startTime=1679219735399; duration=0sec 2023-03-19 09:55:36,238 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:36,625 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/755d481314e64be3947c6299b7bb5408 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/755d481314e64be3947c6299b7bb5408 2023-03-19 09:55:36,628 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72808d88f0cb7335353d6abe6e889fc0/A of 72808d88f0cb7335353d6abe6e889fc0 into 755d481314e64be3947c6299b7bb5408(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-19 09:55:36,628 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:36,628 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0., storeName=72808d88f0cb7335353d6abe6e889fc0/A, priority=11, startTime=1679219735399; duration=1sec 2023-03-19 09:55:36,628 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:39,176 DEBUG [Thread-2433] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:54245 2023-03-19 09:55:39,176 DEBUG [Thread-2433] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:39,228 DEBUG [Thread-2431] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:54245 2023-03-19 09:55:39,228 DEBUG [Thread-2431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:39,870 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-19 09:55:39,907 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f29efcec2bd410ab7c3efeb0636e0a3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7edca62894084a01b9d3126d9721cf64, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ae4a0cc2a2bb4b139f429e33318b8706, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/242d946e910b4dd3b6aff246d44ec31a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/53d48a0a0978445aaeeda13b1caca22e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/fe4c955079244c7fbaebf538fa159e13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/e69be05be9af414ea5bec370d929ec71, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0ddd938ec64641c0ad65e1741961f77a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/35d8092d4c4846439945164aecaf602b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/1a6499105e2944e6ace173741ba7c903, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ccc19a3307cf4f09bc11d5fefcd22f1e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/028385b0402c4cc39b7c386d1305c2ad, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7aa5165f543341f8a49e2063f8fee5e0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a938530e4e7343d486d01e55480f7226, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/b7721c75c6e34a5c9dc03ea1ca8cfe57, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0d3ebbc11c954386bd77a4656f1c1551, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/09bfdcd29ef942e8a2d88f2f349c411b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/cee03ba7b8714648b45b555b0154c8b9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/bb30057c3ef441c49094ff7af9630fa5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a92416197b9a4c22a3f2dbb4881883ab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/dbe95b71aea74e9b9829eb7e5ef8e647, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/5884fff876594dd590b54588e3811df8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f269d372a97424ca9efee6c69b11107, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/79b4830af76a4ff3a597a7273d28e1d1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f0de3be3f884366820ed375703a6e30, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0358470fe776458d93724693dafd55cc] to archive 2023-03-19 09:55:39,908 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:55:39,909 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f29efcec2bd410ab7c3efeb0636e0a3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f29efcec2bd410ab7c3efeb0636e0a3 2023-03-19 09:55:39,910 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7edca62894084a01b9d3126d9721cf64 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7edca62894084a01b9d3126d9721cf64 2023-03-19 09:55:39,911 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ae4a0cc2a2bb4b139f429e33318b8706 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ae4a0cc2a2bb4b139f429e33318b8706 2023-03-19 09:55:39,911 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/242d946e910b4dd3b6aff246d44ec31a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/242d946e910b4dd3b6aff246d44ec31a 2023-03-19 09:55:39,912 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/53d48a0a0978445aaeeda13b1caca22e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/53d48a0a0978445aaeeda13b1caca22e 2023-03-19 09:55:39,913 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/fe4c955079244c7fbaebf538fa159e13 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/fe4c955079244c7fbaebf538fa159e13 2023-03-19 09:55:39,914 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/e69be05be9af414ea5bec370d929ec71 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/e69be05be9af414ea5bec370d929ec71 2023-03-19 09:55:39,914 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0ddd938ec64641c0ad65e1741961f77a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0ddd938ec64641c0ad65e1741961f77a 2023-03-19 09:55:39,915 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/35d8092d4c4846439945164aecaf602b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/35d8092d4c4846439945164aecaf602b 2023-03-19 09:55:39,916 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/1a6499105e2944e6ace173741ba7c903 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/1a6499105e2944e6ace173741ba7c903 2023-03-19 09:55:39,917 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ccc19a3307cf4f09bc11d5fefcd22f1e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/ccc19a3307cf4f09bc11d5fefcd22f1e 2023-03-19 09:55:39,918 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/028385b0402c4cc39b7c386d1305c2ad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/028385b0402c4cc39b7c386d1305c2ad 2023-03-19 09:55:39,918 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7aa5165f543341f8a49e2063f8fee5e0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/7aa5165f543341f8a49e2063f8fee5e0 2023-03-19 09:55:39,919 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a938530e4e7343d486d01e55480f7226 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a938530e4e7343d486d01e55480f7226 2023-03-19 09:55:39,920 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/b7721c75c6e34a5c9dc03ea1ca8cfe57 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/b7721c75c6e34a5c9dc03ea1ca8cfe57 2023-03-19 09:55:39,921 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0d3ebbc11c954386bd77a4656f1c1551 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0d3ebbc11c954386bd77a4656f1c1551 2023-03-19 09:55:39,922 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/09bfdcd29ef942e8a2d88f2f349c411b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/09bfdcd29ef942e8a2d88f2f349c411b 2023-03-19 09:55:39,922 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/cee03ba7b8714648b45b555b0154c8b9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/cee03ba7b8714648b45b555b0154c8b9 2023-03-19 09:55:39,923 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/bb30057c3ef441c49094ff7af9630fa5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/bb30057c3ef441c49094ff7af9630fa5 2023-03-19 09:55:39,924 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a92416197b9a4c22a3f2dbb4881883ab to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/a92416197b9a4c22a3f2dbb4881883ab 2023-03-19 09:55:39,925 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/dbe95b71aea74e9b9829eb7e5ef8e647 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/dbe95b71aea74e9b9829eb7e5ef8e647 2023-03-19 09:55:39,926 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/5884fff876594dd590b54588e3811df8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/5884fff876594dd590b54588e3811df8 2023-03-19 09:55:39,927 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f269d372a97424ca9efee6c69b11107 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f269d372a97424ca9efee6c69b11107 2023-03-19 09:55:39,927 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/79b4830af76a4ff3a597a7273d28e1d1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/79b4830af76a4ff3a597a7273d28e1d1 2023-03-19 09:55:39,928 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f0de3be3f884366820ed375703a6e30 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/2f0de3be3f884366820ed375703a6e30 2023-03-19 09:55:39,929 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0358470fe776458d93724693dafd55cc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/0358470fe776458d93724693dafd55cc 2023-03-19 09:55:39,934 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/877b817f7c9f479b9d693cd9d18e8dc7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e91db938a24f4e4dad7d373eb1b3db4b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/6b33d9430658402e832a13a736180394, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/08a0fc7a4fff42e6b29c423e5a7d7e58, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b11fe9ea218e46a6bd0eee837797e3c5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/def4815270f8461f990ca77ca568e311, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/7e63dab504874ca2b1dc2f4137919efa, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/23f3ed0e84d745589e7b64d98d13f26c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/58b937ab762647c6935416bc04d15fa4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/04250a74ded84801a93a8e09eeacd189, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/77bcda66465f4b1aac7a5691daa5043f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/0916f8a2db04465ca10ecf5e966ce28d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/d44ed5a7235f45a48f339f3e50f05b02, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/67c2cc5fd48d40968583713f3220e4df, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/acfa223d07d24f93b17499b7e20eb3c4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b433e5ba08994e068aadf9b8b358b208, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/9cbf1caa573040b2a902073251c47296, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c1b7436161dc4f289d0f792647172845, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/839dcc6ef13a422fa08ff79a4cb36552, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49078cadf14e4a4f9f9baa26a28b3eed, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/aa051808a7e944a5be162518d63d3aba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/72e7f29815d54edbaa80d4846b22f464, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/296214c92c9746779c205bba6ca9da2d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e4cc5d3b132a4af59854adaef0e8edab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/cee6390a8aa64642bf0071ac8c54a966] to archive 2023-03-19 09:55:39,935 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:55:39,936 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49e32ce7c0ba4e4f9d1b86aff0b98bc9 2023-03-19 09:55:39,937 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/877b817f7c9f479b9d693cd9d18e8dc7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/877b817f7c9f479b9d693cd9d18e8dc7 2023-03-19 09:55:39,937 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e91db938a24f4e4dad7d373eb1b3db4b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e91db938a24f4e4dad7d373eb1b3db4b 2023-03-19 09:55:39,938 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/6b33d9430658402e832a13a736180394 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/6b33d9430658402e832a13a736180394 2023-03-19 09:55:39,939 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/08a0fc7a4fff42e6b29c423e5a7d7e58 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/08a0fc7a4fff42e6b29c423e5a7d7e58 2023-03-19 09:55:39,940 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b11fe9ea218e46a6bd0eee837797e3c5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b11fe9ea218e46a6bd0eee837797e3c5 2023-03-19 09:55:39,941 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/def4815270f8461f990ca77ca568e311 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/def4815270f8461f990ca77ca568e311 2023-03-19 09:55:39,942 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/7e63dab504874ca2b1dc2f4137919efa to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/7e63dab504874ca2b1dc2f4137919efa 2023-03-19 09:55:39,943 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/23f3ed0e84d745589e7b64d98d13f26c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/23f3ed0e84d745589e7b64d98d13f26c 2023-03-19 09:55:39,944 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/58b937ab762647c6935416bc04d15fa4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/58b937ab762647c6935416bc04d15fa4 2023-03-19 09:55:39,945 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/04250a74ded84801a93a8e09eeacd189 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/04250a74ded84801a93a8e09eeacd189 2023-03-19 09:55:39,946 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/77bcda66465f4b1aac7a5691daa5043f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/77bcda66465f4b1aac7a5691daa5043f 2023-03-19 09:55:39,946 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/0916f8a2db04465ca10ecf5e966ce28d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/0916f8a2db04465ca10ecf5e966ce28d 2023-03-19 09:55:39,947 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/d44ed5a7235f45a48f339f3e50f05b02 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/d44ed5a7235f45a48f339f3e50f05b02 2023-03-19 09:55:39,948 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/67c2cc5fd48d40968583713f3220e4df to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/67c2cc5fd48d40968583713f3220e4df 2023-03-19 09:55:39,949 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/acfa223d07d24f93b17499b7e20eb3c4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/acfa223d07d24f93b17499b7e20eb3c4 2023-03-19 09:55:39,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b433e5ba08994e068aadf9b8b358b208 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/b433e5ba08994e068aadf9b8b358b208 2023-03-19 09:55:39,951 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/9cbf1caa573040b2a902073251c47296 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/9cbf1caa573040b2a902073251c47296 2023-03-19 09:55:39,952 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c1b7436161dc4f289d0f792647172845 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c1b7436161dc4f289d0f792647172845 2023-03-19 09:55:39,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/839dcc6ef13a422fa08ff79a4cb36552 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/839dcc6ef13a422fa08ff79a4cb36552 2023-03-19 09:55:39,954 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49078cadf14e4a4f9f9baa26a28b3eed to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/49078cadf14e4a4f9f9baa26a28b3eed 2023-03-19 09:55:39,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/aa051808a7e944a5be162518d63d3aba to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/aa051808a7e944a5be162518d63d3aba 2023-03-19 09:55:39,957 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/72e7f29815d54edbaa80d4846b22f464 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/72e7f29815d54edbaa80d4846b22f464 2023-03-19 09:55:39,958 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/296214c92c9746779c205bba6ca9da2d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/296214c92c9746779c205bba6ca9da2d 2023-03-19 09:55:39,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e4cc5d3b132a4af59854adaef0e8edab to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/e4cc5d3b132a4af59854adaef0e8edab 2023-03-19 09:55:39,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/cee6390a8aa64642bf0071ac8c54a966 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/cee6390a8aa64642bf0071ac8c54a966 2023-03-19 09:55:39,964 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c91ea9c716df4c12a9afd0fb11e59b55, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4c42768b47bb442b80edf39e3f70da85, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/943adc0fcc3f40219f9d7bb534d337ea, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/314ff6fe849147fb99676ccddb0a9e48, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f66d25ce6db8451f8a12490570f7e3f9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/0597f22f2bb44612be633c6b39f3f1b0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/94258e3146594ee39b338f284e00de02, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f746664af2974d64854abd3c1f67e79d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/79d31b64c91e4e4e9ffe5d61605ffaf5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/52a0058af9324dd889d0a28255e6f888, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c5b56997b6c64ae28b7bd13136ea6983, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89ac7ccc6dff42d4a1b05f40899cd500, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/132fc3a1addb4ba988424333705dabc5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/3171f4d5cea04934bb797e7e6a7cd79c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e4e22edee69f4d3992768ff2d96563de, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e30a305cda764bebac61188bd7a65ba7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89f0008cf3084237963067ed8ac7e6e2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4db88d4dbd604eafb5633fcef0dd6a29, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/49450e8ec2ab48c4b7a03966de46c9b6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/43d5acc597c344ceb611e6f4e79a5c11, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6de192fdc8bd422da2850821761cb885, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/324adb2b20f84b50b30920098f1f8d8a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/2882637e009b4063919264423fd9bb6e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/005fd474cd1d47cebaacc48c3faa01dc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/b26b53c6fc7f4570a25f09daca91706f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/fd696cdb187e4cd99dad67efacc902b9] to archive 2023-03-19 09:55:39,965 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:55:39,966 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c91ea9c716df4c12a9afd0fb11e59b55 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c91ea9c716df4c12a9afd0fb11e59b55 2023-03-19 09:55:39,967 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4c42768b47bb442b80edf39e3f70da85 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4c42768b47bb442b80edf39e3f70da85 2023-03-19 09:55:39,968 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/943adc0fcc3f40219f9d7bb534d337ea to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/943adc0fcc3f40219f9d7bb534d337ea 2023-03-19 09:55:39,969 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/314ff6fe849147fb99676ccddb0a9e48 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/314ff6fe849147fb99676ccddb0a9e48 2023-03-19 09:55:39,970 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f66d25ce6db8451f8a12490570f7e3f9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f66d25ce6db8451f8a12490570f7e3f9 2023-03-19 09:55:39,970 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/0597f22f2bb44612be633c6b39f3f1b0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/0597f22f2bb44612be633c6b39f3f1b0 2023-03-19 09:55:39,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/94258e3146594ee39b338f284e00de02 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/94258e3146594ee39b338f284e00de02 2023-03-19 09:55:39,972 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f746664af2974d64854abd3c1f67e79d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/f746664af2974d64854abd3c1f67e79d 2023-03-19 09:55:39,972 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/79d31b64c91e4e4e9ffe5d61605ffaf5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/79d31b64c91e4e4e9ffe5d61605ffaf5 2023-03-19 09:55:39,973 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/52a0058af9324dd889d0a28255e6f888 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/52a0058af9324dd889d0a28255e6f888 2023-03-19 09:55:39,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c5b56997b6c64ae28b7bd13136ea6983 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/c5b56997b6c64ae28b7bd13136ea6983 2023-03-19 09:55:39,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89ac7ccc6dff42d4a1b05f40899cd500 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89ac7ccc6dff42d4a1b05f40899cd500 2023-03-19 09:55:39,975 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/132fc3a1addb4ba988424333705dabc5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/132fc3a1addb4ba988424333705dabc5 2023-03-19 09:55:39,976 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/3171f4d5cea04934bb797e7e6a7cd79c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/3171f4d5cea04934bb797e7e6a7cd79c 2023-03-19 09:55:39,977 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e4e22edee69f4d3992768ff2d96563de to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e4e22edee69f4d3992768ff2d96563de 2023-03-19 09:55:39,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e30a305cda764bebac61188bd7a65ba7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/e30a305cda764bebac61188bd7a65ba7 2023-03-19 09:55:39,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89f0008cf3084237963067ed8ac7e6e2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/89f0008cf3084237963067ed8ac7e6e2 2023-03-19 09:55:39,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4db88d4dbd604eafb5633fcef0dd6a29 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/4db88d4dbd604eafb5633fcef0dd6a29 2023-03-19 09:55:39,981 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/49450e8ec2ab48c4b7a03966de46c9b6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/49450e8ec2ab48c4b7a03966de46c9b6 2023-03-19 09:55:39,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/43d5acc597c344ceb611e6f4e79a5c11 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/43d5acc597c344ceb611e6f4e79a5c11 2023-03-19 09:55:39,983 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6de192fdc8bd422da2850821761cb885 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6de192fdc8bd422da2850821761cb885 2023-03-19 09:55:39,984 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/324adb2b20f84b50b30920098f1f8d8a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/324adb2b20f84b50b30920098f1f8d8a 2023-03-19 09:55:39,985 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/2882637e009b4063919264423fd9bb6e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/2882637e009b4063919264423fd9bb6e 2023-03-19 09:55:39,986 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/005fd474cd1d47cebaacc48c3faa01dc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/005fd474cd1d47cebaacc48c3faa01dc 2023-03-19 09:55:39,987 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/b26b53c6fc7f4570a25f09daca91706f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/b26b53c6fc7f4570a25f09daca91706f 2023-03-19 09:55:39,988 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/fd696cdb187e4cd99dad67efacc902b9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/fd696cdb187e4cd99dad67efacc902b9 2023-03-19 09:55:42,932 DEBUG [Thread-2437] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:54245 2023-03-19 09:55:42,932 DEBUG [Thread-2437] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:42,941 DEBUG [Thread-2435] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:54245 2023-03-19 09:55:42,941 DEBUG [Thread-2435] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:42,941 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-19 09:55:42,941 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 240 2023-03-19 09:55:42,941 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-19 09:55:42,941 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1843 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5529 rows 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1826 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5478 rows 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1817 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5446 rows 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1822 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5460 rows 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1836 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5508 rows 2023-03-19 09:55:42,942 INFO [Listener at localhost/45961] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-19 09:55:42,942 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:54245 2023-03-19 09:55:42,942 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:55:42,943 INFO [Listener at localhost/45961] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-19 09:55:42,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-19 09:55:42,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:42,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-19 09:55:42,948 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219742948"}]},"ts":"1679219742948"} 2023-03-19 09:55:42,949 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-19 09:55:42,953 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-19 09:55:42,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, UNASSIGN}] 2023-03-19 09:55:42,955 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, UNASSIGN 2023-03-19 09:55:42,956 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:42,956 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219742956"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219742956"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219742956"}]},"ts":"1679219742956"} 2023-03-19 09:55:42,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:55:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-19 09:55:43,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:43,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 72808d88f0cb7335353d6abe6e889fc0, disabling compactions & flushes 2023-03-19 09:55:43,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:43,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:43,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. after waiting 0 ms 2023-03-19 09:55:43,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:43,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 72808d88f0cb7335353d6abe6e889fc0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-19 09:55:43,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=A 2023-03-19 09:55:43,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:43,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=B 2023-03-19 09:55:43,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:43,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72808d88f0cb7335353d6abe6e889fc0, store=C 2023-03-19 09:55:43,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:43,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-19 09:55:43,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319ad4f9ae8cba5479f8e0b305337e37b5a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ad4f9ae8cba5479f8e0b305337e37b5a 2023-03-19 09:55:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-19 09:55:43,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=635, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/72854bac507f4277b24f94e2374f9632 2023-03-19 09:55:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-19 09:55:44,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/c58849afef794bc5affc3ac0fd737290 2023-03-19 09:55:44,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/6ce32687eda3494ea7d1bbd4117a2ced 2023-03-19 09:55:44,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/A/72854bac507f4277b24f94e2374f9632 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/72854bac507f4277b24f94e2374f9632 2023-03-19 09:55:44,747 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/72854bac507f4277b24f94e2374f9632, entries=150, sequenceid=635, filesize=25.4 K 2023-03-19 09:55:44,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/B/c58849afef794bc5affc3ac0fd737290 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c58849afef794bc5affc3ac0fd737290 2023-03-19 09:55:44,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c58849afef794bc5affc3ac0fd737290, entries=150, sequenceid=635, filesize=11.9 K 2023-03-19 09:55:44,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/.tmp/C/6ce32687eda3494ea7d1bbd4117a2ced as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6ce32687eda3494ea7d1bbd4117a2ced 2023-03-19 09:55:44,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6ce32687eda3494ea7d1bbd4117a2ced, entries=150, sequenceid=635, filesize=11.9 K 2023-03-19 09:55:44,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 72808d88f0cb7335353d6abe6e889fc0 in 1647ms, sequenceid=635, compaction requested=false 2023-03-19 09:55:44,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/recovered.edits/638.seqid, newMaxSeqId=638, maxSeqId=4 2023-03-19 09:55:44,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0. 2023-03-19 09:55:44,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 72808d88f0cb7335353d6abe6e889fc0: 2023-03-19 09:55:44,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:44,764 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=72808d88f0cb7335353d6abe6e889fc0, regionState=CLOSED 2023-03-19 09:55:44,764 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219744764"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219744764"}]},"ts":"1679219744764"} 2023-03-19 09:55:44,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-19 09:55:44,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 72808d88f0cb7335353d6abe6e889fc0, server=jenkins-hbase4.apache.org,34669,1679219618625 in 1.8090 sec 2023-03-19 09:55:44,767 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-19 09:55:44,767 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=72808d88f0cb7335353d6abe6e889fc0, UNASSIGN in 1.8120 sec 2023-03-19 09:55:44,767 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219744767"}]},"ts":"1679219744767"} 2023-03-19 09:55:44,768 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-19 09:55:44,769 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-19 09:55:44,771 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8260 sec 2023-03-19 09:55:45,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-19 09:55:45,052 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-19 09:55:45,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-19 09:55:45,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:45,055 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:45,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-19 09:55:45,055 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:45,059 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:45,061 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/recovered.edits] 2023-03-19 09:55:45,063 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/72854bac507f4277b24f94e2374f9632 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/72854bac507f4277b24f94e2374f9632 2023-03-19 09:55:45,064 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/755d481314e64be3947c6299b7bb5408 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/A/755d481314e64be3947c6299b7bb5408 2023-03-19 09:55:45,066 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/065290ce7ec848748d4652eacd9373ed to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/065290ce7ec848748d4652eacd9373ed 2023-03-19 09:55:45,067 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c58849afef794bc5affc3ac0fd737290 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/B/c58849afef794bc5affc3ac0fd737290 2023-03-19 09:55:45,069 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6ce32687eda3494ea7d1bbd4117a2ced to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/6ce32687eda3494ea7d1bbd4117a2ced 2023-03-19 09:55:45,070 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/7724f0ad48e04dc9978f945a92baa3c8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/C/7724f0ad48e04dc9978f945a92baa3c8 2023-03-19 09:55:45,071 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/recovered.edits/638.seqid to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0/recovered.edits/638.seqid 2023-03-19 09:55:45,072 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/72808d88f0cb7335353d6abe6e889fc0 2023-03-19 09:55:45,072 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:55:45,072 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-19 09:55:45,073 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-19 09:55:45,075 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303191ca788eb48ad4766b2db97a465cb5b30 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303191ca788eb48ad4766b2db97a465cb5b30 2023-03-19 09:55:45,075 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031933afde41a16f4b8892d0de556bc26cb6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031933afde41a16f4b8892d0de556bc26cb6 2023-03-19 09:55:45,077 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031955db6534faef4cd988dcfef93628c610 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031955db6534faef4cd988dcfef93628c610 2023-03-19 09:55:45,077 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195aee43a7e3c9430ca2dd4146a591c070 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195aee43a7e3c9430ca2dd4146a591c070 2023-03-19 09:55:45,078 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195d63d30ab2cc4d1e9a90c02818a9939f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195d63d30ab2cc4d1e9a90c02818a9939f 2023-03-19 09:55:45,079 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319694399355ff144f6b09d20eba5137a8a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319694399355ff144f6b09d20eba5137a8a 2023-03-19 09:55:45,080 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196f59f45e6ee04ffcb1f461e695714e89 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196f59f45e6ee04ffcb1f461e695714e89 2023-03-19 09:55:45,081 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319828dcc19538e4953b5dc86b1aafea699 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319828dcc19538e4953b5dc86b1aafea699 2023-03-19 09:55:45,082 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198bcf2a0da469414390800aaba082a07e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198bcf2a0da469414390800aaba082a07e 2023-03-19 09:55:45,082 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a1e608bcf8974995b6b8e219f206375c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a1e608bcf8974995b6b8e219f206375c 2023-03-19 09:55:45,083 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a748fe262c0b4e30ba652a0c67aaf095 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a748fe262c0b4e30ba652a0c67aaf095 2023-03-19 09:55:45,084 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ac0b21d521d9435eb7394644de16980f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ac0b21d521d9435eb7394644de16980f 2023-03-19 09:55:45,085 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ad4f9ae8cba5479f8e0b305337e37b5a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ad4f9ae8cba5479f8e0b305337e37b5a 2023-03-19 09:55:45,086 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b4a00ffabc8d453c9e022b5430ec6d0f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b4a00ffabc8d453c9e022b5430ec6d0f 2023-03-19 09:55:45,086 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bbd0f55fd400419daa9c4e9e4ed64551 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bbd0f55fd400419daa9c4e9e4ed64551 2023-03-19 09:55:45,087 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bc226da026c0475a8105eb5c0c489a2f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bc226da026c0475a8105eb5c0c489a2f 2023-03-19 09:55:45,088 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bd9f584c2c4e4c8588bb4711b29794d7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319bd9f584c2c4e4c8588bb4711b29794d7 2023-03-19 09:55:45,089 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319d42ae1f7fee14d288533ea717342befd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319d42ae1f7fee14d288533ea717342befd 2023-03-19 09:55:45,090 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319db8592f895e84ad7a5d9a9a660460296 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319db8592f895e84ad7a5d9a9a660460296 2023-03-19 09:55:45,091 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e32314887b4b4d62ba88dde25cc763c1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319e32314887b4b4d62ba88dde25cc763c1 2023-03-19 09:55:45,092 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319f66ec662f6a04e808fd646da50ecaafb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319f66ec662f6a04e808fd646da50ecaafb 2023-03-19 09:55:45,092 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-19 09:55:45,094 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:45,095 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-19 09:55:45,096 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-19 09:55:45,097 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:45,097 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-19 09:55:45,097 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679219745097"}]},"ts":"9223372036854775807"} 2023-03-19 09:55:45,099 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-19 09:55:45,099 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 72808d88f0cb7335353d6abe6e889fc0, NAME => 'TestAcidGuarantees,,1679219710389.72808d88f0cb7335353d6abe6e889fc0.', STARTKEY => '', ENDKEY => ''}] 2023-03-19 09:55:45,099 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-19 09:55:45,099 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679219745099"}]},"ts":"9223372036854775807"} 2023-03-19 09:55:45,100 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-19 09:55:45,101 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:45,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 49 msec 2023-03-19 09:55:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-19 09:55:45,156 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-19 09:55:45,167 INFO [Listener at localhost/45961] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 227), OpenFileDescriptor=507 (was 559), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 405), ProcessCount=169 (was 169), AvailableMemoryMB=1386 (was 1536) 2023-03-19 09:55:45,179 INFO [Listener at localhost/45961] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=507, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=169, AvailableMemoryMB=1386 2023-03-19 09:55:45,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:55:45,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:55:45,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-19 09:55:45,183 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-19 09:55:45,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-19 09:55:45,183 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-19 09:55:45,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-19 09:55:45,184 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,185 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731 empty. 2023-03-19 09:55:45,185 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,185 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:55:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-19 09:55:45,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-19 09:55:45,592 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:55:45,593 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ddd80e01130df7b3867030aa3a103731, NAME => 'TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp 2023-03-19 09:55:45,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:55:45,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ddd80e01130df7b3867030aa3a103731, disabling compactions & flushes 2023-03-19 09:55:45,605 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. after waiting 0 ms 2023-03-19 09:55:45,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,606 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,606 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:45,608 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-19 09:55:45,608 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219745608"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219745608"}]},"ts":"1679219745608"} 2023-03-19 09:55:45,609 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-19 09:55:45,610 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-19 09:55:45,610 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219745610"}]},"ts":"1679219745610"} 2023-03-19 09:55:45,611 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-19 09:55:45,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ddd80e01130df7b3867030aa3a103731, ASSIGN}] 2023-03-19 09:55:45,617 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ddd80e01130df7b3867030aa3a103731, ASSIGN 2023-03-19 09:55:45,617 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ddd80e01130df7b3867030aa3a103731, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=false 2023-03-19 09:55:45,768 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=ddd80e01130df7b3867030aa3a103731, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:45,768 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219745768"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219745768"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219745768"}]},"ts":"1679219745768"} 2023-03-19 09:55:45,770 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:55:45,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-19 09:55:45,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ddd80e01130df7b3867030aa3a103731, NAME => 'TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:55:45,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:55:45,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,924 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,925 DEBUG [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A 2023-03-19 09:55:45,926 DEBUG [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A 2023-03-19 09:55:45,926 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:45,926 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ddd80e01130df7b3867030aa3a103731 columnFamilyName A 2023-03-19 09:55:45,926 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.HStore(310): Store=ddd80e01130df7b3867030aa3a103731/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:45,926 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,928 DEBUG [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B 2023-03-19 09:55:45,928 DEBUG [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B 2023-03-19 09:55:45,928 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:45,928 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ddd80e01130df7b3867030aa3a103731 columnFamilyName B 2023-03-19 09:55:45,928 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.HStore(310): Store=ddd80e01130df7b3867030aa3a103731/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:45,929 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,929 DEBUG [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C 2023-03-19 09:55:45,929 DEBUG [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C 2023-03-19 09:55:45,930 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:55:45,930 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ddd80e01130df7b3867030aa3a103731 columnFamilyName C 2023-03-19 09:55:45,930 INFO [StoreOpener-ddd80e01130df7b3867030aa3a103731-1] regionserver.HStore(310): Store=ddd80e01130df7b3867030aa3a103731/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:55:45,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:55:45,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:45,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:55:45,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ddd80e01130df7b3867030aa3a103731; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71194485, jitterRate=0.060880497097969055}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:55:45,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:45,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., pid=49, masterSystemTime=1679219745921 2023-03-19 09:55:45,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:45,937 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=ddd80e01130df7b3867030aa3a103731, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:45,937 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219745937"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219745937"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219745937"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219745937"}]},"ts":"1679219745937"} 2023-03-19 09:55:45,939 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-19 09:55:45,939 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 in 168 msec 2023-03-19 09:55:45,941 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-19 09:55:45,941 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ddd80e01130df7b3867030aa3a103731, ASSIGN in 324 msec 2023-03-19 09:55:45,941 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-19 09:55:45,941 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219745941"}]},"ts":"1679219745941"} 2023-03-19 09:55:45,942 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-19 09:55:45,944 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-19 09:55:45,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 763 msec 2023-03-19 09:55:46,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-19 09:55:46,287 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-19 09:55:46,288 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x02df63a4 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,293 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@501c5a88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,294 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,296 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48214, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,297 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-19 09:55:46,299 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51656, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-19 09:55:46,300 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,304 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,304 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,309 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,310 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,318 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,318 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,323 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,323 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,329 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,329 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x461f5f3f to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,336 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426cb82a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,336 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,343 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,343 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,347 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,348 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,354 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,354 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:55:46,362 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:55:46,376 DEBUG [hconnection-0x7461910d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,376 DEBUG [hconnection-0x6d22e10e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,377 DEBUG [hconnection-0x4f77690b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,378 DEBUG [hconnection-0xeae7bd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,378 DEBUG [hconnection-0x2a7665ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,378 DEBUG [hconnection-0x347c8b06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,379 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48226, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,384 DEBUG [hconnection-0x24e3d1b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,384 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48224, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,384 DEBUG [hconnection-0x30a871e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,385 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48242, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,390 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48244, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,390 DEBUG [hconnection-0x323b2a4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,390 DEBUG [hconnection-0xce26c93-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:55:46,390 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48228, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,391 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48286, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:46,400 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,400 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48270, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,401 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48288, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,401 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48300, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:55:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:46,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:46,411 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:46,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:46,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:46,412 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:46,415 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,415 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,415 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,415 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:46,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:46,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:55:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:46,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:46,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:46,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:46,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:46,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:46,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:46,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:46,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:46,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,424 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:46,424 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:46,426 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ccc242c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:46,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:46,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,430 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:46,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:46,435 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219806434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219806434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219806435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219806435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219806436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:46,438 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:46,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:46,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:46,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:46,438 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:46,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,444 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:46,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:46,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,445 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,445 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,445 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,445 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,445 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:46,445 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:46,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/27bcd395e3a64d15a1a78a33179d03cd 2023-03-19 09:55:46,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219806537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219806537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219806537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219806537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219806537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,545 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:46,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:46,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:46,551 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:46,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:46,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:46,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:46,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:46,559 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,559 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:46,559 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:46,559 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:46,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:46,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:46,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119f73eb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:46,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:46,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,564 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:46,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,564 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:46,566 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:46,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:46,568 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:46,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:46,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:46,568 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:46,568 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:46,572 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,572 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,572 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,573 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:46,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:46,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:46,575 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,575 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,576 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:46,576 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:46,676 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:46,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:46,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:46,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:46,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:46,679 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:46,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:46,679 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:46,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,681 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:46,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:46,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:46,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:46,683 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,683 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:46,683 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:46,683 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:46,685 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7448557d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:46,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:46,686 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,688 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:46,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,688 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:46,688 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,690 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,690 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:46,692 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:46,692 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:46,692 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:46,692 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:46,693 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:46,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,694 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:46,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:46,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,698 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,698 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,699 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:46,699 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:46,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219806741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219806741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219806742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219806742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:46,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219806742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,799 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:46,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:46,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:46,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:46,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:46,802 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:46,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:46,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:46,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,805 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,805 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:46,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:46,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:46,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:46,806 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:46,806 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:46,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:46,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658baeeb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:46,811 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:46,811 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,812 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:46,812 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:46,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,812 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:46,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:46,813 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:46,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:46,816 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:46,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:46,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:46,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:46,816 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:46,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,820 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:46,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,821 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:46,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:46,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,825 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:46,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:46,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:46,825 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:46,825 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:46,825 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:46,825 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,825 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:46,825 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,825 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:46,825 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/282a5e7b714947528b59c738c01601db 2023-03-19 09:55:46,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/c3f28305197f47b88ef375a5c59707da 2023-03-19 09:55:46,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/27bcd395e3a64d15a1a78a33179d03cd as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/27bcd395e3a64d15a1a78a33179d03cd 2023-03-19 09:55:46,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/27bcd395e3a64d15a1a78a33179d03cd, entries=150, sequenceid=16, filesize=11.6 K 2023-03-19 09:55:46,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/282a5e7b714947528b59c738c01601db as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/282a5e7b714947528b59c738c01601db 2023-03-19 09:55:46,889 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/282a5e7b714947528b59c738c01601db, entries=150, sequenceid=16, filesize=11.6 K 2023-03-19 09:55:46,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/c3f28305197f47b88ef375a5c59707da as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c3f28305197f47b88ef375a5c59707da 2023-03-19 09:55:46,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c3f28305197f47b88ef375a5c59707da, entries=150, sequenceid=16, filesize=11.6 K 2023-03-19 09:55:46,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for ddd80e01130df7b3867030aa3a103731 in 473ms, sequenceid=16, compaction requested=false 2023-03-19 09:55:46,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:46,925 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:46,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:46,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:46,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:46,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:46,930 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:46,930 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,930 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:46,930 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:46,931 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,931 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,932 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:46,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:46,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:46,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:46,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:46,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:46,933 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:46,933 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:46,933 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:55:46,934 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:46,934 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:46,934 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:46,934 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:46,934 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:46,934 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:47,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:47,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:47,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219807053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219807054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219807057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219807054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219807058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219807164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219807164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219807165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219807165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219807165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,342 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/65c07ca9ab6b4aafbc9bfda3cc98c95c 2023-03-19 09:55:47,354 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/6d41cd64864244138ea80f37f228a815 2023-03-19 09:55:47,367 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/664253bc14ce4a09a3dacb08524eec06 2023-03-19 09:55:47,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219807370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219807371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,373 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/65c07ca9ab6b4aafbc9bfda3cc98c95c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/65c07ca9ab6b4aafbc9bfda3cc98c95c 2023-03-19 09:55:47,376 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/65c07ca9ab6b4aafbc9bfda3cc98c95c, entries=150, sequenceid=34, filesize=11.6 K 2023-03-19 09:55:47,377 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/6d41cd64864244138ea80f37f228a815 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6d41cd64864244138ea80f37f228a815 2023-03-19 09:55:47,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219807373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219807373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,380 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6d41cd64864244138ea80f37f228a815, entries=150, sequenceid=34, filesize=11.6 K 2023-03-19 09:55:47,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219807374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,381 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/664253bc14ce4a09a3dacb08524eec06 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/664253bc14ce4a09a3dacb08524eec06 2023-03-19 09:55:47,386 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/664253bc14ce4a09a3dacb08524eec06, entries=150, sequenceid=34, filesize=11.6 K 2023-03-19 09:55:47,387 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ddd80e01130df7b3867030aa3a103731 in 454ms, sequenceid=34, compaction requested=false 2023-03-19 09:55:47,387 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:47,387 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,388 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:47,388 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:47,389 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,389 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:47,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:47,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@585b1b32[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:47,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:47,391 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,392 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:47,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:47,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:47,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:47,398 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:47,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:47,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:47,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:47,398 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:47,401 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,401 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:47,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,401 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:47,401 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:47,402 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:47,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,404 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:47,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:47,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:47,405 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:47,404 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,405 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:47,405 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:47,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:47,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:47,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:47,505 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:47,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:47,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:47,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:47,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:47,508 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:47,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:47,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:47,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:47,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:47,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:47,510 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:47,511 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,511 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:47,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:47,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:47,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:47,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:47,512 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:47,512 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:47,512 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-19 09:55:47,513 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:47,513 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:47,513 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:47,513 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:47,513 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:47,513 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:47,528 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/0b256e14282448d3b46bc6a96944e568 2023-03-19 09:55:47,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:47,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219807694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219807694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219807695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219807695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219807696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219807797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219807802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219807802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219807802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219807802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:47,938 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/642b2a3042ed42f7a78634790d36f2f4 2023-03-19 09:55:48,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219808004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219808007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219808007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219808007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219808007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219808310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219808315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219808315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219808315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219808317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,355 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5cb128a8f6804a4b90a1e53bb95a3c29 2023-03-19 09:55:48,360 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/0b256e14282448d3b46bc6a96944e568 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0b256e14282448d3b46bc6a96944e568 2023-03-19 09:55:48,364 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0b256e14282448d3b46bc6a96944e568, entries=150, sequenceid=48, filesize=11.6 K 2023-03-19 09:55:48,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/642b2a3042ed42f7a78634790d36f2f4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/642b2a3042ed42f7a78634790d36f2f4 2023-03-19 09:55:48,368 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/642b2a3042ed42f7a78634790d36f2f4, entries=150, sequenceid=48, filesize=11.6 K 2023-03-19 09:55:48,369 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5cb128a8f6804a4b90a1e53bb95a3c29 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5cb128a8f6804a4b90a1e53bb95a3c29 2023-03-19 09:55:48,377 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5cb128a8f6804a4b90a1e53bb95a3c29, entries=150, sequenceid=48, filesize=11.6 K 2023-03-19 09:55:48,377 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ddd80e01130df7b3867030aa3a103731 in 865ms, sequenceid=48, compaction requested=true 2023-03-19 09:55:48,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:48,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:48,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:48,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:48,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:48,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:48,379 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,379 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:48,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:48,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b12bf9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:48,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:48,381 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:48,383 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,383 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:48,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:48,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:48,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:48,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:48,388 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:48,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:48,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:48,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:48,388 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:48,390 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,390 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:48,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,390 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:48,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:48,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:48,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:48,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:48,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:48,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:48,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:48,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:48,395 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:48,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:48,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,395 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:48,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:48,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,395 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:48,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:48,495 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:48,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:48,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:48,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:48,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:48,498 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:48,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:48,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:48,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:48,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:48,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,501 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:48,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:48,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:48,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:48,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:48,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:48,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:48,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:48,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:48,502 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:48,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:48,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:48,503 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:48,503 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:48,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:48,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:48,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219808837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219808838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219808838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219808839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219808839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,910 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/a517eb2b9f0e4b389e7c7c8a82165255 2023-03-19 09:55:48,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219808940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219808945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219808948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219808952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:48,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219808952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219809147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219809153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219809158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219809158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219809159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,319 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/61c29e43303c4825aae200cc720c0c31 2023-03-19 09:55:49,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219809453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219809461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219809471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219809471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:49,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219809471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,728 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/a7341af58e8d465ebc6e332a2a85f642 2023-03-19 09:55:49,732 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/a517eb2b9f0e4b389e7c7c8a82165255 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a517eb2b9f0e4b389e7c7c8a82165255 2023-03-19 09:55:49,735 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a517eb2b9f0e4b389e7c7c8a82165255, entries=150, sequenceid=66, filesize=11.6 K 2023-03-19 09:55:49,736 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/61c29e43303c4825aae200cc720c0c31 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/61c29e43303c4825aae200cc720c0c31 2023-03-19 09:55:49,738 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/61c29e43303c4825aae200cc720c0c31, entries=150, sequenceid=66, filesize=11.6 K 2023-03-19 09:55:49,739 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/a7341af58e8d465ebc6e332a2a85f642 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a7341af58e8d465ebc6e332a2a85f642 2023-03-19 09:55:49,742 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a7341af58e8d465ebc6e332a2a85f642, entries=150, sequenceid=66, filesize=11.6 K 2023-03-19 09:55:49,742 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for ddd80e01130df7b3867030aa3a103731 in 1239ms, sequenceid=66, compaction requested=true 2023-03-19 09:55:49,742 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:49,742 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:49,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:49,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:49,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464736f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:49,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:49,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,748 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:49,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,748 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:49,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:49,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:49,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:49,752 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:49,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:49,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:49,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:49,752 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:49,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:49,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,756 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:49,757 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:49,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:49,761 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:49,761 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:49,762 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:49,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:49,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:49,862 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:49,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:49,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:49,865 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:49,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:49,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:49,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:49,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:49,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:49,866 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:49,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:49,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:49,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:49,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:49,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:49,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:49,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:49,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:49,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:49,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:49,869 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:49,869 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:49,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:49,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:50,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219810005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219810012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219810013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219810014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219810015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219810116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219810133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219810133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219810133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219810133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,278 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/bd1a801f80a442608e603c434511ba67 2023-03-19 09:55:50,287 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-19 09:55:50,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219810322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219810336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219810336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219810336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219810336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219810629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219810642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219810643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219810643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:50,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219810643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,691 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/9d11d4a41d7c4fbd957cd74881f4f01f 2023-03-19 09:55:50,702 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/987be4fe581c46dfa6cad18801800b5a 2023-03-19 09:55:50,706 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/bd1a801f80a442608e603c434511ba67 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bd1a801f80a442608e603c434511ba67 2023-03-19 09:55:50,709 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bd1a801f80a442608e603c434511ba67, entries=150, sequenceid=81, filesize=11.6 K 2023-03-19 09:55:50,709 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/9d11d4a41d7c4fbd957cd74881f4f01f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9d11d4a41d7c4fbd957cd74881f4f01f 2023-03-19 09:55:50,712 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9d11d4a41d7c4fbd957cd74881f4f01f, entries=150, sequenceid=81, filesize=11.6 K 2023-03-19 09:55:50,713 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/987be4fe581c46dfa6cad18801800b5a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/987be4fe581c46dfa6cad18801800b5a 2023-03-19 09:55:50,715 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/987be4fe581c46dfa6cad18801800b5a, entries=150, sequenceid=81, filesize=11.6 K 2023-03-19 09:55:50,716 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ddd80e01130df7b3867030aa3a103731 in 847ms, sequenceid=81, compaction requested=true 2023-03-19 09:55:50,716 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:50,716 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:50,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:50,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:50,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:50,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:50,721 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:50,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:50,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c2079fe[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:50,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:50,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,724 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:50,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:50,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:50,726 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:50,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:50,728 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:50,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:50,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:50,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:50,728 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:50,730 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,730 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:50,730 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:50,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:50,731 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:50,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:50,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:50,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:50,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:50,736 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:50,736 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:50,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:50,836 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:50,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:50,846 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:50,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:50,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:50,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:50,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:50,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:50,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:50,849 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:50,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:50,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:50,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:50,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:50,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:50,860 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:50,860 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:50,860 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:50,860 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:50,861 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:55:50,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:50,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:50,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:50,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:50,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:50,861 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:50,868 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/83825c3d5a8a4e2eae50cfc904a9e10d 2023-03-19 09:55:51,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:51,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:51,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219811162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219811163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219811163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219811169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219811169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,280 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/7f12c03d9f9544628570a43f2f71ecec 2023-03-19 09:55:51,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219811275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219811275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219811277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219811277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219811277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219811481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219811481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219811482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219811484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219811484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,697 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/2e7630b1417d42fabb7ee4cca2be77f3 2023-03-19 09:55:51,701 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/83825c3d5a8a4e2eae50cfc904a9e10d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/83825c3d5a8a4e2eae50cfc904a9e10d 2023-03-19 09:55:51,704 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/83825c3d5a8a4e2eae50cfc904a9e10d, entries=150, sequenceid=98, filesize=11.6 K 2023-03-19 09:55:51,705 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/7f12c03d9f9544628570a43f2f71ecec as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/7f12c03d9f9544628570a43f2f71ecec 2023-03-19 09:55:51,707 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/7f12c03d9f9544628570a43f2f71ecec, entries=150, sequenceid=98, filesize=11.6 K 2023-03-19 09:55:51,708 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/2e7630b1417d42fabb7ee4cca2be77f3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2e7630b1417d42fabb7ee4cca2be77f3 2023-03-19 09:55:51,711 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2e7630b1417d42fabb7ee4cca2be77f3, entries=150, sequenceid=98, filesize=11.6 K 2023-03-19 09:55:51,711 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ddd80e01130df7b3867030aa3a103731 in 850ms, sequenceid=98, compaction requested=true 2023-03-19 09:55:51,712 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:51,712 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,714 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:51,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca267c3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:51,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:51,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:51,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:51,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,719 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:51,719 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:51,719 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:51,722 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:51,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:51,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:51,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:51,722 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:51,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:51,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,726 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:51,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:51,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:51,734 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:51,734 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:51,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:51,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:51,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:51,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:51,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:51,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:55:51,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:51,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:51,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:51,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:51,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:51,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:51,803 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/333950f9edb54f1582a38841a023418f 2023-03-19 09:55:51,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219811816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219811816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219811817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219811818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,834 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:51,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:51,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219811827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:51,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:51,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:51,837 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:51,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:51,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:51,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:51,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,837 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,838 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:51,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:51,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:51,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:51,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:51,840 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:51,840 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:51,840 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:51,840 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:51,843 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:51,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f77d4d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:51,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:51,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,847 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:51,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:51,848 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:51,851 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:51,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:51,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:51,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:51,851 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:51,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:51,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:51,866 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:51,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:51,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,866 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,866 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:51,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:51,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219811928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219811931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219811931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219811931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:51,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219811935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,967 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:51,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:51,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:51,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:51,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:51,970 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:51,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:51,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:51,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:51,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:51,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:51,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:51,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:51,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:51,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:51,974 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:51,974 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:51,974 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:51,974 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:51,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:51,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202f89d7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:51,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:51,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,979 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,979 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:51,979 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,979 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:51,979 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:51,979 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:51,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:51,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:51,984 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:51,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:51,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:51,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:51,984 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:51,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:51,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,987 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:51,987 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,988 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:51,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:51,990 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:51,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:51,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:51,990 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:51,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,991 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:51,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:51,991 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:51,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:51,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:51,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:51,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,091 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:52,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:52,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:52,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:52,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:52,096 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:52,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:52,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:52,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,098 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:52,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:52,100 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,100 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:52,100 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:52,101 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:52,104 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,104 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,104 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:52,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7feba5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:52,106 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:52,106 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:52,108 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,108 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:52,112 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:52,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:52,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:52,115 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:52,115 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:52,115 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:52,115 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:52,115 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:52,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:52,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:52,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:52,131 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,132 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:52,132 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:52,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219812134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219812139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219812139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219812139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219812143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,222 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/f99b200ef61a42b2a66c91db28023e29 2023-03-19 09:55:52,232 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:52,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:52,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:52,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:52,235 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:52,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:52,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:52,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:52,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:52,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:52,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:52,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,239 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:52,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:52,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:52,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:52,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:52,240 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:52,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:52,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:52,240 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:52,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:52,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@304c1bf6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:52,245 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:52,245 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,246 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:52,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:52,246 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:52,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:52,248 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:52,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:52,250 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:52,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:52,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:52,250 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:52,250 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:52,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,252 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,252 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,253 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:52,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:52,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,257 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,257 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:52,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,257 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:52,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,257 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,357 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:52,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:52,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:52,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:52,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:52,363 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:52,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:52,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:52,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:52,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:52,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:52,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:52,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:52,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:52,368 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,369 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:52,369 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,373 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:52,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ced15a7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:52,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:52,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:52,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:52,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:52,378 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,378 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:52,379 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:52,379 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,379 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,379 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:52,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:52,382 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:52,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:52,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:52,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:52,382 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:52,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,390 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,392 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,392 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,392 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:52,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:52,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,394 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:52,394 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:52,392 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,394 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219812443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219812446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219812446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219812446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219812452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,494 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:52,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:52,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:52,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:52,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:52,497 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:52,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:52,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:52,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,499 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,499 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:52,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:52,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:52,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:52,501 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:52,503 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:52,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d054dc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:52,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:52,505 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:52,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:52,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:52,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:52,511 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:52,512 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:52,512 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:52,512 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:52,512 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:52,513 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,513 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,513 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,514 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,515 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,518 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:52,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,518 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,518 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:52,518 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:52,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,619 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:52,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:52,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:52,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:52,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:52,622 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:52,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:52,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:52,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,624 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:52,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:52,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:52,626 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,626 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:52,626 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:52,626 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:52,627 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,627 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3092afa0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:52,630 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:52,630 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,632 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:52,632 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,632 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:52,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:52,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,633 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,633 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/2b09132417294074a630abd3707aed60 2023-03-19 09:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:52,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:52,638 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:52,638 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:52,638 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:52,638 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:52,638 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:52,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/333950f9edb54f1582a38841a023418f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/333950f9edb54f1582a38841a023418f 2023-03-19 09:55:52,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,642 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:52,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:52,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/333950f9edb54f1582a38841a023418f, entries=150, sequenceid=115, filesize=11.6 K 2023-03-19 09:55:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,644 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/f99b200ef61a42b2a66c91db28023e29 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f99b200ef61a42b2a66c91db28023e29 2023-03-19 09:55:52,644 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,646 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:52,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:52,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:52,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:52,647 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:52,647 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:52,647 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:52,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:52,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f99b200ef61a42b2a66c91db28023e29, entries=150, sequenceid=115, filesize=11.6 K 2023-03-19 09:55:52,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/2b09132417294074a630abd3707aed60 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2b09132417294074a630abd3707aed60 2023-03-19 09:55:52,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2b09132417294074a630abd3707aed60, entries=150, sequenceid=115, filesize=11.6 K 2023-03-19 09:55:52,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ddd80e01130df7b3867030aa3a103731 in 857ms, sequenceid=115, compaction requested=true 2023-03-19 09:55:52,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:52,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:52,652 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-19 09:55:52,652 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-19 09:55:52,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:52,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:52,654 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-19 09:55:52,654 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:55:52,654 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,654 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-19 09:55:52,654 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/282a5e7b714947528b59c738c01601db, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6d41cd64864244138ea80f37f228a815, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/642b2a3042ed42f7a78634790d36f2f4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/61c29e43303c4825aae200cc720c0c31, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9d11d4a41d7c4fbd957cd74881f4f01f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/7f12c03d9f9544628570a43f2f71ecec, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f99b200ef61a42b2a66c91db28023e29] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=81.3 K 2023-03-19 09:55:52,654 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:55:52,654 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,654 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/27bcd395e3a64d15a1a78a33179d03cd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/65c07ca9ab6b4aafbc9bfda3cc98c95c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0b256e14282448d3b46bc6a96944e568, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a517eb2b9f0e4b389e7c7c8a82165255, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bd1a801f80a442608e603c434511ba67, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/83825c3d5a8a4e2eae50cfc904a9e10d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/333950f9edb54f1582a38841a023418f] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=81.3 K 2023-03-19 09:55:52,655 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 282a5e7b714947528b59c738c01601db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679219746414 2023-03-19 09:55:52,655 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 27bcd395e3a64d15a1a78a33179d03cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679219746414 2023-03-19 09:55:52,655 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6d41cd64864244138ea80f37f228a815, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679219746432 2023-03-19 09:55:52,655 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 65c07ca9ab6b4aafbc9bfda3cc98c95c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679219746432 2023-03-19 09:55:52,656 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 642b2a3042ed42f7a78634790d36f2f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679219747052 2023-03-19 09:55:52,656 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0b256e14282448d3b46bc6a96944e568, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679219747052 2023-03-19 09:55:52,656 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 61c29e43303c4825aae200cc720c0c31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679219747681 2023-03-19 09:55:52,656 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a517eb2b9f0e4b389e7c7c8a82165255, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679219747681 2023-03-19 09:55:52,656 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9d11d4a41d7c4fbd957cd74881f4f01f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679219748836 2023-03-19 09:55:52,656 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting bd1a801f80a442608e603c434511ba67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679219748836 2023-03-19 09:55:52,657 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 7f12c03d9f9544628570a43f2f71ecec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679219749981 2023-03-19 09:55:52,657 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 83825c3d5a8a4e2eae50cfc904a9e10d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679219749981 2023-03-19 09:55:52,657 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f99b200ef61a42b2a66c91db28023e29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679219751167 2023-03-19 09:55:52,657 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 333950f9edb54f1582a38841a023418f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679219751167 2023-03-19 09:55:52,667 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:52,669 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#424 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:52,747 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:52,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:52,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:52,750 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:52,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:52,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:52,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,752 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:52,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:52,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:52,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:52,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:52,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:52,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:52,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:52,754 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:52,754 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:52,754 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-19 09:55:52,754 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:52,754 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:52,754 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:52,754 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:52,754 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:52,762 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/0462a3e58d384d3d822df3c852b929e3 2023-03-19 09:55:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:52,961 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:52,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219812976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219812981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219812982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219812983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:52,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:52,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219812986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,077 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/e1178b6f43064569a58fd44ea5bfb350 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e1178b6f43064569a58fd44ea5bfb350 2023-03-19 09:55:53,082 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/2a522ecaa06a4fd6a8816cfd756d17ec as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/2a522ecaa06a4fd6a8816cfd756d17ec 2023-03-19 09:55:53,083 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into e1178b6f43064569a58fd44ea5bfb350(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:53,083 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,083 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=9, startTime=1679219752652; duration=0sec 2023-03-19 09:55:53,083 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:53,083 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-19 09:55:53,085 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-19 09:55:53,085 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:55:53,085 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,086 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c3f28305197f47b88ef375a5c59707da, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/664253bc14ce4a09a3dacb08524eec06, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5cb128a8f6804a4b90a1e53bb95a3c29, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a7341af58e8d465ebc6e332a2a85f642, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/987be4fe581c46dfa6cad18801800b5a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2e7630b1417d42fabb7ee4cca2be77f3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2b09132417294074a630abd3707aed60] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=81.3 K 2023-03-19 09:55:53,086 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c3f28305197f47b88ef375a5c59707da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679219746414 2023-03-19 09:55:53,086 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into 2a522ecaa06a4fd6a8816cfd756d17ec(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:53,086 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,086 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=9, startTime=1679219752652; duration=0sec 2023-03-19 09:55:53,086 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:53,086 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 664253bc14ce4a09a3dacb08524eec06, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679219746432 2023-03-19 09:55:53,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5cb128a8f6804a4b90a1e53bb95a3c29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679219747052 2023-03-19 09:55:53,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a7341af58e8d465ebc6e332a2a85f642, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679219747681 2023-03-19 09:55:53,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 987be4fe581c46dfa6cad18801800b5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679219748836 2023-03-19 09:55:53,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2e7630b1417d42fabb7ee4cca2be77f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679219749981 2023-03-19 09:55:53,088 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2b09132417294074a630abd3707aed60, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679219751167 2023-03-19 09:55:53,096 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#427 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:53,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219813087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219813091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219813091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219813093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219813096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,103 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/747727899350441b84aca9f64d12af5a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/747727899350441b84aca9f64d12af5a 2023-03-19 09:55:53,106 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into 747727899350441b84aca9f64d12af5a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:53,106 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,106 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=9, startTime=1679219752652; duration=0sec 2023-03-19 09:55:53,106 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:53,171 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/424adecf0bc44bd982fae95b80238113 2023-03-19 09:55:53,181 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/b74e3da5c11f4aa8b7904de967f97c73 2023-03-19 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/0462a3e58d384d3d822df3c852b929e3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0462a3e58d384d3d822df3c852b929e3 2023-03-19 09:55:53,188 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0462a3e58d384d3d822df3c852b929e3, entries=150, sequenceid=130, filesize=11.7 K 2023-03-19 09:55:53,189 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/424adecf0bc44bd982fae95b80238113 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/424adecf0bc44bd982fae95b80238113 2023-03-19 09:55:53,192 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/424adecf0bc44bd982fae95b80238113, entries=150, sequenceid=130, filesize=11.7 K 2023-03-19 09:55:53,193 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/b74e3da5c11f4aa8b7904de967f97c73 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/b74e3da5c11f4aa8b7904de967f97c73 2023-03-19 09:55:53,196 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/b74e3da5c11f4aa8b7904de967f97c73, entries=150, sequenceid=130, filesize=11.7 K 2023-03-19 09:55:53,197 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ddd80e01130df7b3867030aa3a103731 in 443ms, sequenceid=130, compaction requested=false 2023-03-19 09:55:53,197 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,197 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:53,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605181ef[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:53,201 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:53,201 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,205 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:53,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:53,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:53,210 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:53,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:53,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:53,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:53,210 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:53,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,212 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,218 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:53,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:53,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,218 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:53,219 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:53,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,219 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:53,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:55:53,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:53,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:53,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:53,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,319 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:53,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/b67934882a2a44e9a4563cffcd1ea5ba 2023-03-19 09:55:53,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:53,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:53,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219813319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219813322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219813323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219813326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219813326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:53,336 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:53,336 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,336 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:53,336 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:53,336 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,337 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,338 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,338 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:53,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:53,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:53,343 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,343 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:53,343 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:53,343 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:53,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:53,346 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:53,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f5f06f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:53,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:53,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:53,350 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,350 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:53,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:53,355 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:53,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:53,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:53,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:53,356 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:53,358 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,358 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,358 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,358 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,359 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,362 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,362 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:53,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:53,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,362 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,362 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:53,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219813427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219813429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219813430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219813435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219813437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,462 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:53,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:53,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:53,466 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:53,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:53,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:53,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,467 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:53,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:53,469 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,469 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:53,469 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:53,469 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:53,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,472 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:53,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d13299[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:53,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:53,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,476 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:53,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,478 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:53,480 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:53,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:53,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:53,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:53,480 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:53,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,484 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,487 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,487 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:53,488 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:53,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,588 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:53,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:53,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:53,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:53,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:53,591 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:53,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:53,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:53,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,593 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:53,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:53,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:53,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,597 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,597 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731b784c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:53,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:53,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,599 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:53,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:53,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:53,605 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,605 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:53,607 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:53,607 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:53,607 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:53,607 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:53,607 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:53,609 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,609 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,609 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,609 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,610 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:53,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:53,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,613 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:53,613 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:53,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219813631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219813634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219813634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219813640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219813641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,713 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:53,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:53,719 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:53,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:53,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:53,720 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,720 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:53,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:53,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:53,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:53,728 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,728 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:53,728 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:53,728 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,731 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f29d139[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:53,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:53,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:53,739 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,739 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:53,744 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:53,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:53,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:53,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:53,744 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:53,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/35ea1663b3d44e12b4c9033ac9fc6019 2023-03-19 09:55:53,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,752 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,754 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:53,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:53,755 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:53,755 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:53,764 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/94848d579c5741898d05fb5502d9b248 2023-03-19 09:55:53,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/b67934882a2a44e9a4563cffcd1ea5ba as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/b67934882a2a44e9a4563cffcd1ea5ba 2023-03-19 09:55:53,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/b67934882a2a44e9a4563cffcd1ea5ba, entries=300, sequenceid=153, filesize=18.8 K 2023-03-19 09:55:53,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/35ea1663b3d44e12b4c9033ac9fc6019 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/35ea1663b3d44e12b4c9033ac9fc6019 2023-03-19 09:55:53,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/35ea1663b3d44e12b4c9033ac9fc6019, entries=150, sequenceid=153, filesize=11.8 K 2023-03-19 09:55:53,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/94848d579c5741898d05fb5502d9b248 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/94848d579c5741898d05fb5502d9b248 2023-03-19 09:55:53,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/94848d579c5741898d05fb5502d9b248, entries=150, sequenceid=153, filesize=11.8 K 2023-03-19 09:55:53,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for ddd80e01130df7b3867030aa3a103731 in 474ms, sequenceid=153, compaction requested=true 2023-03-19 09:55:53,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,779 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:53,779 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:53,779 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:53,779 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:53,779 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:53,780 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43268 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:53,781 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:55:53,781 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,781 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:53,781 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/2a522ecaa06a4fd6a8816cfd756d17ec, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0462a3e58d384d3d822df3c852b929e3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/b67934882a2a44e9a4563cffcd1ea5ba] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=42.3 K 2023-03-19 09:55:53,781 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:55:53,781 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,781 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e1178b6f43064569a58fd44ea5bfb350, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/424adecf0bc44bd982fae95b80238113, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/35ea1663b3d44e12b4c9033ac9fc6019] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=35.3 K 2023-03-19 09:55:53,781 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2a522ecaa06a4fd6a8816cfd756d17ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679219751167 2023-03-19 09:55:53,781 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e1178b6f43064569a58fd44ea5bfb350, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679219751167 2023-03-19 09:55:53,781 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0462a3e58d384d3d822df3c852b929e3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1679219751814 2023-03-19 09:55:53,782 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 424adecf0bc44bd982fae95b80238113, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1679219751814 2023-03-19 09:55:53,782 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b67934882a2a44e9a4563cffcd1ea5ba, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679219752974 2023-03-19 09:55:53,782 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 35ea1663b3d44e12b4c9033ac9fc6019, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679219752974 2023-03-19 09:55:53,789 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:53,789 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#433 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:53,805 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/bc522fd8485341a0a45166da5990063a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bc522fd8485341a0a45166da5990063a 2023-03-19 09:55:53,808 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into bc522fd8485341a0a45166da5990063a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:53,809 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,809 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=13, startTime=1679219753779; duration=0sec 2023-03-19 09:55:53,809 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:53,809 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:53,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:53,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:55:53,810 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,810 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/747727899350441b84aca9f64d12af5a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/b74e3da5c11f4aa8b7904de967f97c73, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/94848d579c5741898d05fb5502d9b248] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=35.3 K 2023-03-19 09:55:53,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 747727899350441b84aca9f64d12af5a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679219751167 2023-03-19 09:55:53,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b74e3da5c11f4aa8b7904de967f97c73, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1679219751814 2023-03-19 09:55:53,811 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 94848d579c5741898d05fb5502d9b248, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679219752974 2023-03-19 09:55:53,837 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#434 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:53,851 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/4135c80123694cff9c2b1ebb60009c9f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4135c80123694cff9c2b1ebb60009c9f 2023-03-19 09:55:53,854 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into 4135c80123694cff9c2b1ebb60009c9f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:53,854 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,854 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=13, startTime=1679219753779; duration=0sec 2023-03-19 09:55:53,855 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:53,855 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:53,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:53,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:53,858 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:53,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:53,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:53,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,859 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:53,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:53,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:53,862 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,862 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:53,863 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:55:53,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:53,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:53,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:53,863 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,874 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/f68cb769799341bc8c19c6ebc14f6499 2023-03-19 09:55:53,883 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/fdd47ff413ac41c5a6c86f19eeb20dba 2023-03-19 09:55:53,892 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/935eade305d94ac081f537ca89adb054 2023-03-19 09:55:53,897 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/f68cb769799341bc8c19c6ebc14f6499 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f68cb769799341bc8c19c6ebc14f6499 2023-03-19 09:55:53,902 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f68cb769799341bc8c19c6ebc14f6499, entries=150, sequenceid=168, filesize=11.8 K 2023-03-19 09:55:53,903 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/fdd47ff413ac41c5a6c86f19eeb20dba as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/fdd47ff413ac41c5a6c86f19eeb20dba 2023-03-19 09:55:53,909 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/fdd47ff413ac41c5a6c86f19eeb20dba, entries=150, sequenceid=168, filesize=11.8 K 2023-03-19 09:55:53,911 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/935eade305d94ac081f537ca89adb054 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/935eade305d94ac081f537ca89adb054 2023-03-19 09:55:53,919 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/935eade305d94ac081f537ca89adb054, entries=150, sequenceid=168, filesize=11.8 K 2023-03-19 09:55:53,928 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for ddd80e01130df7b3867030aa3a103731 in 66ms, sequenceid=168, compaction requested=false 2023-03-19 09:55:53,928 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:53,928 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:53,928 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:53,930 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,930 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,931 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,931 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34f42c3b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:53,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:53,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,937 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:53,937 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,937 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:53,941 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:53,941 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,941 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,941 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:53,944 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:53,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:53,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:53,944 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:53,944 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:53,946 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,946 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:53,946 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,947 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:53,948 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:53,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:53,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:53,951 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:53,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:53,951 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:53,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:53,952 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:53,952 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:53,952 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:53,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:55:53,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:53,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:53,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:53,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:53,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219814009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219814010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219754005, max timestamp=1679219754021Num uniques 859; 2023-03-19 09:55:54,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219814010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:54,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219754005, max timestamp=1679219754021Num uniques 859; 2023-03-19 09:55:54,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:54,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219754005, max timestamp=1679219754021Num uniques 859; 2023-03-19 09:55:54,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:54,052 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:54,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:54,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:54,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:54,057 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:54,057 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,057 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:54,057 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:54,058 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,058 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,059 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,060 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,060 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:54,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:54,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:54,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:54,072 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:54,075 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6e85c6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:54,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:54,076 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,078 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:54,078 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,078 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:54,079 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:54,082 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:54,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:54,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:54,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:54,082 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:54,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219814073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,084 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,084 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,084 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,084 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,085 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,091 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,091 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:54,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:54,091 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,091 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:54,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,091 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,091 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:54,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219814083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219814124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219814124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219814124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219814184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,192 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:54,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:54,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:54,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:54,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:54,195 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219814193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,197 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:54,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:54,201 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,203 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/c750a5733b764f6e94dd4e2cd7ba6426 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/c750a5733b764f6e94dd4e2cd7ba6426 2023-03-19 09:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:54,203 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@431f49ff[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:54,203 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,208 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into c750a5733b764f6e94dd4e2cd7ba6426(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:54,208 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:54,208 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=13, startTime=1679219753779; duration=0sec 2023-03-19 09:55:54,208 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:54,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:54,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:54,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:54,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:54,213 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:54,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:54,215 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:54,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:54,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:54,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:54,215 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:54,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,218 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,219 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,222 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,222 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,222 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:54,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:54,222 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,223 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:54,223 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:54,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,323 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:54,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:54,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:54,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:54,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:54,326 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:54,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:54,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:54,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,328 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,330 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,330 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,331 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,331 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:54,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:54,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:54,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:54,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219814330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219814331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,335 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:54,335 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:54,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219814332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:54,336 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:54,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:54,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,337 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:54,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12eee8bc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:54,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,340 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:54,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:54,341 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:54,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,342 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:54,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:54,346 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:54,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:54,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:54,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:54,346 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:54,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,351 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:54,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:54,354 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:54,354 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:54,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/16ff2bcb855e4076bbe9a1302c9a2e42 2023-03-19 09:55:54,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219814390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219814399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,454 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:54,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:54,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:54,463 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:54,463 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:54,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:54,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,464 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,465 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,465 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,466 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:54,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:54,467 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,467 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:54,467 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:54,467 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:54,467 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,469 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e9b2e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:54,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:54,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,473 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:54,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:54,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:54,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:54,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:54,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:54,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:54,482 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:54,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:54,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:54,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:54,482 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:54,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,493 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,510 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:54,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:54,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,511 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:54,511 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:54,510 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,511 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,511 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,511 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,611 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:54,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:54,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:54,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:54,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:54,614 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:54,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:54,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:54,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,617 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,617 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,618 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:54,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:54,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:54,619 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,619 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:54,619 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:54,619 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:54,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:54,623 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18317eb0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:54,623 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,626 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:54,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,626 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,628 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,628 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:54,631 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:54,631 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:54,631 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:54,631 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:54,631 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:54,632 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,633 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,633 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,634 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,637 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,637 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:54,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:54,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:54,637 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,637 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:54,637 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:54,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,638 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219814636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219814637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219814638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219814703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:54,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219814706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,737 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:54,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:54,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:54,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:54,740 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:54,740 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:54,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:54,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,744 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:54,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:54,745 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,745 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:54,745 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:54,745 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:54,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21588cf8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:54,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:54,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,750 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:54,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,750 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:54,752 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:54,754 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:54,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:54,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:54,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:54,754 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:54,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:54,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,759 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:54,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:54,762 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:54,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:54,763 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:54,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,763 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:54,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:54,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:54,763 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/9e6ef826264046829b55458822bc9b0c 2023-03-19 09:55:54,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/f938e509ba9640028b40ca4b850592d9 2023-03-19 09:55:54,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/16ff2bcb855e4076bbe9a1302c9a2e42 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/16ff2bcb855e4076bbe9a1302c9a2e42 2023-03-19 09:55:54,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/16ff2bcb855e4076bbe9a1302c9a2e42, entries=150, sequenceid=179, filesize=11.8 K 2023-03-19 09:55:54,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/9e6ef826264046829b55458822bc9b0c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9e6ef826264046829b55458822bc9b0c 2023-03-19 09:55:54,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9e6ef826264046829b55458822bc9b0c, entries=150, sequenceid=179, filesize=11.8 K 2023-03-19 09:55:54,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/f938e509ba9640028b40ca4b850592d9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/f938e509ba9640028b40ca4b850592d9 2023-03-19 09:55:54,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/f938e509ba9640028b40ca4b850592d9, entries=150, sequenceid=179, filesize=11.8 K 2023-03-19 09:55:54,854 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ddd80e01130df7b3867030aa3a103731 in 882ms, sequenceid=179, compaction requested=true 2023-03-19 09:55:54,854 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:54,854 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:54,854 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:54,854 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:54,854 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:54,854 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:54,855 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:54,855 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:55:54,855 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:54,855 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,855 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:55:54,855 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bc522fd8485341a0a45166da5990063a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f68cb769799341bc8c19c6ebc14f6499, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/16ff2bcb855e4076bbe9a1302c9a2e42] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=35.6 K 2023-03-19 09:55:54,855 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,856 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/c750a5733b764f6e94dd4e2cd7ba6426, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/fdd47ff413ac41c5a6c86f19eeb20dba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9e6ef826264046829b55458822bc9b0c] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=35.6 K 2023-03-19 09:55:54,856 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting bc522fd8485341a0a45166da5990063a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679219752974 2023-03-19 09:55:54,857 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c750a5733b764f6e94dd4e2cd7ba6426, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679219752974 2023-03-19 09:55:54,857 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting f68cb769799341bc8c19c6ebc14f6499, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1679219753318 2023-03-19 09:55:54,858 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fdd47ff413ac41c5a6c86f19eeb20dba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1679219753318 2023-03-19 09:55:54,858 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 16ff2bcb855e4076bbe9a1302c9a2e42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679219753950 2023-03-19 09:55:54,858 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9e6ef826264046829b55458822bc9b0c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679219753950 2023-03-19 09:55:54,863 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:54,864 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:54,864 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:54,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:54,866 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:54,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:54,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:54,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,868 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,868 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:54,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:54,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:54,869 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:54,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:54,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:54,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:54,871 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,871 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:54,872 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-19 09:55:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:54,877 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/3c3e2a9df2f64bc5a52c70b3b6a32525 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3c3e2a9df2f64bc5a52c70b3b6a32525 2023-03-19 09:55:54,882 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/4e851d5d99c14c4e965333c47d303dab as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e851d5d99c14c4e965333c47d303dab 2023-03-19 09:55:54,885 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into 3c3e2a9df2f64bc5a52c70b3b6a32525(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:54,885 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into 4e851d5d99c14c4e965333c47d303dab(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:54,885 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:54,886 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:54,886 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=13, startTime=1679219754854; duration=0sec 2023-03-19 09:55:54,886 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=13, startTime=1679219754854; duration=0sec 2023-03-19 09:55:54,886 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:54,886 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:54,886 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:55:54,886 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:55:54,887 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:55:54,887 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:54,887 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4135c80123694cff9c2b1ebb60009c9f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/935eade305d94ac081f537ca89adb054, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/f938e509ba9640028b40ca4b850592d9] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=35.6 K 2023-03-19 09:55:54,887 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 4135c80123694cff9c2b1ebb60009c9f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679219752974 2023-03-19 09:55:54,887 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 935eade305d94ac081f537ca89adb054, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1679219753318 2023-03-19 09:55:54,888 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting f938e509ba9640028b40ca4b850592d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679219753950 2023-03-19 09:55:54,899 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#444 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:54,923 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/d7f8b1f873974c4fb1e309c38775375c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d7f8b1f873974c4fb1e309c38775375c 2023-03-19 09:55:54,928 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into d7f8b1f873974c4fb1e309c38775375c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:54,928 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:54,928 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=13, startTime=1679219754854; duration=0sec 2023-03-19 09:55:54,928 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:55,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:55,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219815162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219815163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219815164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219815209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219815217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219815267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219815272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219815276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,295 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/3ebba2eff88f4a7f90dc422dec3561c0 2023-03-19 09:55:55,305 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/2db4a1e5321b40a6b5da876092dead76 2023-03-19 09:55:55,316 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/773d22000f894dd688b94505b413119a 2023-03-19 09:55:55,320 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/3ebba2eff88f4a7f90dc422dec3561c0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ebba2eff88f4a7f90dc422dec3561c0 2023-03-19 09:55:55,323 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ebba2eff88f4a7f90dc422dec3561c0, entries=150, sequenceid=216, filesize=11.8 K 2023-03-19 09:55:55,324 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/2db4a1e5321b40a6b5da876092dead76 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2db4a1e5321b40a6b5da876092dead76 2023-03-19 09:55:55,327 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2db4a1e5321b40a6b5da876092dead76, entries=150, sequenceid=216, filesize=11.8 K 2023-03-19 09:55:55,328 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/773d22000f894dd688b94505b413119a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/773d22000f894dd688b94505b413119a 2023-03-19 09:55:55,332 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/773d22000f894dd688b94505b413119a, entries=150, sequenceid=216, filesize=11.8 K 2023-03-19 09:55:55,333 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for ddd80e01130df7b3867030aa3a103731 in 461ms, sequenceid=216, compaction requested=false 2023-03-19 09:55:55,333 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:55,333 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:55,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:55,337 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:55,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44163b67[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:55,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:55,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,341 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:55,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:55,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:55,345 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:55,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:55,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:55,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:55,346 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:55,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:55,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:55,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:55,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,352 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,352 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:55,352 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:55,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:55,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:55,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:55,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:55,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:55,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:55,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:55,353 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:55,353 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:55,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:55,453 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:55,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:55,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:55,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:55,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:55,456 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:55,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:55,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:55,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:55,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:55,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:55,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:55,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:55,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:55,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:55,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:55,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:55,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:55,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:55,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:55,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:55,463 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:55,465 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:55:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:55,465 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:55,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:55,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:55,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679219755502, max timestamp=1679219755516Num uniques 859; 2023-03-19 09:55:55,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679219755502, max timestamp=1679219755516Num uniques 859; 2023-03-19 09:55:55,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679219755502, max timestamp=1679219755516Num uniques 859; 2023-03-19 09:55:55,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:55:55,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:55:55,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:55:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219815538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219815539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219815540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219815648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219815648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219815649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219815859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219815860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219815860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:55,875 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/04f98b64b1bf4d17b132e93ed6e38970 2023-03-19 09:55:55,884 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/6171c200eb0143a891034fa78fc9e5c3 2023-03-19 09:55:56,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219816165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219816165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48226 deadline: 1679219816166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219816224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219816230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,293 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5d9beae549704ba7b91eb1168d6ddd5f 2023-03-19 09:55:56,298 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/04f98b64b1bf4d17b132e93ed6e38970 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/04f98b64b1bf4d17b132e93ed6e38970 2023-03-19 09:55:56,301 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/04f98b64b1bf4d17b132e93ed6e38970, entries=150, sequenceid=230, filesize=11.8 K 2023-03-19 09:55:56,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/6171c200eb0143a891034fa78fc9e5c3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6171c200eb0143a891034fa78fc9e5c3 2023-03-19 09:55:56,305 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6171c200eb0143a891034fa78fc9e5c3, entries=150, sequenceid=230, filesize=11.8 K 2023-03-19 09:55:56,306 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5d9beae549704ba7b91eb1168d6ddd5f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9beae549704ba7b91eb1168d6ddd5f 2023-03-19 09:55:56,309 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9beae549704ba7b91eb1168d6ddd5f, entries=150, sequenceid=230, filesize=11.8 K 2023-03-19 09:55:56,309 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ddd80e01130df7b3867030aa3a103731 in 844ms, sequenceid=230, compaction requested=true 2023-03-19 09:55:56,309 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:56,309 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,311 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37526f20[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:56,313 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:56,313 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,314 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:56,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:56,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:56,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:56,316 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:56,318 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:56,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:56,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:56,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:56,318 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:56,322 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,322 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:56,322 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:56,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:56,323 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,325 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,325 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:56,325 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:56,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:56,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:56,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:56,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:56,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:56,325 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:56,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,326 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:56,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:56,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:56,326 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:56,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,426 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:56,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:56,429 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:56,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:56,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:56,429 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:56,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:56,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,431 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:56,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:56,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,431 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:56,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:56,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:56,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:56,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:56,433 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:56,433 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:56,433 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-19 09:55:56,433 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:56,433 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:56,433 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:56,434 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:56,434 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:56,434 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:56,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:56,678 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:56,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679219816689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219816690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219816694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219816794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48270 deadline: 1679219816798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219816804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:56,841 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/e1300cb22c68409caa7c6054f118d165 2023-03-19 09:55:57,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219816999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219817006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219817011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,250 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/2df48427866d4f57b4a265b8c742ec41 2023-03-19 09:55:57,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219817305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219817311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219817315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,660 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/8f610fdd4ea344109b948cbaab72b2d6 2023-03-19 09:55:57,663 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/e1300cb22c68409caa7c6054f118d165 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e1300cb22c68409caa7c6054f118d165 2023-03-19 09:55:57,666 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e1300cb22c68409caa7c6054f118d165, entries=150, sequenceid=264, filesize=11.9 K 2023-03-19 09:55:57,666 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/2df48427866d4f57b4a265b8c742ec41 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2df48427866d4f57b4a265b8c742ec41 2023-03-19 09:55:57,670 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2df48427866d4f57b4a265b8c742ec41, entries=150, sequenceid=264, filesize=11.9 K 2023-03-19 09:55:57,670 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/8f610fdd4ea344109b948cbaab72b2d6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8f610fdd4ea344109b948cbaab72b2d6 2023-03-19 09:55:57,673 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8f610fdd4ea344109b948cbaab72b2d6, entries=150, sequenceid=264, filesize=11.9 K 2023-03-19 09:55:57,674 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for ddd80e01130df7b3867030aa3a103731 in 1241ms, sequenceid=264, compaction requested=true 2023-03-19 09:55:57,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:57,674 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:57,676 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,676 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,676 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,676 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24854384[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:57,677 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:57,678 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,679 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:57,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:57,681 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:57,683 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:57,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:57,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:57,683 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:57,683 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:57,685 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,685 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,685 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:57,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,687 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:57,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:57,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:57,689 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:57,689 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:57,689 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:57,790 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:57,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:57,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:57,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:57,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:57,793 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:57,793 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:57,793 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:57,793 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:57,793 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,793 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,796 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:57,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,797 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:57,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:57,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:57,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:57,798 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:57,798 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:57,798 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:57,798 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:57,805 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/a6718236c4d7485b9497655ead87be0e 2023-03-19 09:55:57,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:57,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:57,818 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/8be2dcfb60934af1b05d568baf7c8761 2023-03-19 09:55:57,826 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/34cd6be951dd4571a569d3f6b6757e37 2023-03-19 09:55:57,828 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/a6718236c4d7485b9497655ead87be0e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a6718236c4d7485b9497655ead87be0e 2023-03-19 09:55:57,832 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a6718236c4d7485b9497655ead87be0e, entries=150, sequenceid=275, filesize=11.9 K 2023-03-19 09:55:57,832 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/8be2dcfb60934af1b05d568baf7c8761 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/8be2dcfb60934af1b05d568baf7c8761 2023-03-19 09:55:57,835 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/8be2dcfb60934af1b05d568baf7c8761, entries=150, sequenceid=275, filesize=11.9 K 2023-03-19 09:55:57,836 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/34cd6be951dd4571a569d3f6b6757e37 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/34cd6be951dd4571a569d3f6b6757e37 2023-03-19 09:55:57,839 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/34cd6be951dd4571a569d3f6b6757e37, entries=150, sequenceid=275, filesize=11.9 K 2023-03-19 09:55:57,839 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=40.25 KB/41220 for ddd80e01130df7b3867030aa3a103731 in 41ms, sequenceid=275, compaction requested=true 2023-03-19 09:55:57,839 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:57,839 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:57,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:57,839 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:55:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:57,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:57,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:57,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:57,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:57,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:57,842 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,842 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,842 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,842 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:57,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41df24a0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:57,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:57,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,845 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:57,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:57,847 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:57,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:57,850 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:57,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:57,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:57,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:57,850 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:57,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/6b6dd8912fe2412ebffa4fbd633a8f36 2023-03-19 09:55:57,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,856 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,856 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:57,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:57,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,857 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/f3529f291be44400b04e84212489461c 2023-03-19 09:55:57,859 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:57,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:57,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:57,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,860 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:57,860 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:57,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:57,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679219757875, max timestamp=1679219757901Num uniques 859; 2023-03-19 09:55:57,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679219757875, max timestamp=1679219757901Num uniques 859; 2023-03-19 09:55:57,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679219757875, max timestamp=1679219757901Num uniques 859; 2023-03-19 09:55:57,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:55:57,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:55:57,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-19 09:55:57,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219817931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219817936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:57,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219817941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,960 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:57,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:57,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:57,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:57,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:57,963 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:57,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:57,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:57,963 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:57,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,966 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,966 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:57,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:57,966 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,966 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:57,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:57,967 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:57,967 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:57,967 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:57,967 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:57,969 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e7f2ff[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:57,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:57,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:57,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:57,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:57,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:57,978 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:57,980 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:57,980 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:57,980 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:57,980 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:57,980 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:57,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:57,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,982 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:57,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:57,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:57,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:57,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:57,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:57,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:57,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:57,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:57,987 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:57,987 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:57,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219818048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219818048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219818048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,087 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:58,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:58,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:58,096 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:58,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:58,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:58,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:58,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,096 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,099 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:58,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:58,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:58,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:58,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:58,112 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,112 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:58,112 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:58,112 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:58,112 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:58,114 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:58,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,115 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:58,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:58,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10e8b02c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:58,116 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:58,116 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:58,118 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,118 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:58,119 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:58,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:58,122 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:58,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:58,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:58,122 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:58,122 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:58,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:58,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:58,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:58,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:58,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:58,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:58,129 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:58,129 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:58,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,129 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,129 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:58,229 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:58,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:58,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:58,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:58,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:58,232 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:58,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:58,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:58,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:58,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,234 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:58,234 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:58,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:58,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:58,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:58,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:58,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:58,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:58,236 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:58,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:58,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eff48d6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:58,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:58,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,241 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:58,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:58,242 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:58,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:58,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:58,243 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:58,246 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:58,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:58,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:58,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:58,246 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:58,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219818240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,251 DEBUG [Thread-3188] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:58,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:58,251 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,251 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,253 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,255 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:58,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:58,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219818244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:58,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:58,256 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:58,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:58,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:58,257 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:58,257 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:58,260 DEBUG [Thread-3192] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:55:58,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:58,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219818260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219818260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219818261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/6e605770e2ca453ba8d04ddf59b5df8b 2023-03-19 09:55:58,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/6b6dd8912fe2412ebffa4fbd633a8f36 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6b6dd8912fe2412ebffa4fbd633a8f36 2023-03-19 09:55:58,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6b6dd8912fe2412ebffa4fbd633a8f36, entries=250, sequenceid=285, filesize=16.7 K 2023-03-19 09:55:58,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/f3529f291be44400b04e84212489461c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f3529f291be44400b04e84212489461c 2023-03-19 09:55:58,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f3529f291be44400b04e84212489461c, entries=150, sequenceid=285, filesize=11.9 K 2023-03-19 09:55:58,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/6e605770e2ca453ba8d04ddf59b5df8b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/6e605770e2ca453ba8d04ddf59b5df8b 2023-03-19 09:55:58,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/6e605770e2ca453ba8d04ddf59b5df8b, entries=150, sequenceid=285, filesize=11.9 K 2023-03-19 09:55:58,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ddd80e01130df7b3867030aa3a103731 in 444ms, sequenceid=285, compaction requested=true 2023-03-19 09:55:58,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:58,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:58,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:58,283 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-19 09:55:58,283 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-19 09:55:58,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:58,285 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 77990 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-19 09:55:58,285 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73110 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-19 09:55:58,285 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:55:58,285 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:55:58,285 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,285 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,285 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3c3e2a9df2f64bc5a52c70b3b6a32525, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ebba2eff88f4a7f90dc422dec3561c0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/04f98b64b1bf4d17b132e93ed6e38970, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e1300cb22c68409caa7c6054f118d165, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a6718236c4d7485b9497655ead87be0e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6b6dd8912fe2412ebffa4fbd633a8f36] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=76.2 K 2023-03-19 09:55:58,285 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e851d5d99c14c4e965333c47d303dab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2db4a1e5321b40a6b5da876092dead76, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6171c200eb0143a891034fa78fc9e5c3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2df48427866d4f57b4a265b8c742ec41, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/8be2dcfb60934af1b05d568baf7c8761, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f3529f291be44400b04e84212489461c] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=71.4 K 2023-03-19 09:55:58,286 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3c3e2a9df2f64bc5a52c70b3b6a32525, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679219753950 2023-03-19 09:55:58,286 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3ebba2eff88f4a7f90dc422dec3561c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679219754063 2023-03-19 09:55:58,286 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4e851d5d99c14c4e965333c47d303dab, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679219753950 2023-03-19 09:55:58,286 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 04f98b64b1bf4d17b132e93ed6e38970, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219755155 2023-03-19 09:55:58,286 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2db4a1e5321b40a6b5da876092dead76, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679219754063 2023-03-19 09:55:58,287 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting e1300cb22c68409caa7c6054f118d165, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679219755534 2023-03-19 09:55:58,287 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6171c200eb0143a891034fa78fc9e5c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219755155 2023-03-19 09:55:58,287 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a6718236c4d7485b9497655ead87be0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679219756685 2023-03-19 09:55:58,287 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2df48427866d4f57b4a265b8c742ec41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679219755534 2023-03-19 09:55:58,287 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 6b6dd8912fe2412ebffa4fbd633a8f36, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679219757815 2023-03-19 09:55:58,287 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8be2dcfb60934af1b05d568baf7c8761, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679219756685 2023-03-19 09:55:58,288 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f3529f291be44400b04e84212489461c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679219757830 2023-03-19 09:55:58,297 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#459 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:58,298 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#460 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:58,305 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/d1126312d8ad45eda4b05b75068cf4ee as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d1126312d8ad45eda4b05b75068cf4ee 2023-03-19 09:55:58,310 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into d1126312d8ad45eda4b05b75068cf4ee(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:58,310 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:58,310 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=10, startTime=1679219758283; duration=0sec 2023-03-19 09:55:58,310 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:55:58,310 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-19 09:55:58,311 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73110 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-19 09:55:58,311 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:55:58,311 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,311 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d7f8b1f873974c4fb1e309c38775375c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/773d22000f894dd688b94505b413119a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9beae549704ba7b91eb1168d6ddd5f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8f610fdd4ea344109b948cbaab72b2d6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/34cd6be951dd4571a569d3f6b6757e37, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/6e605770e2ca453ba8d04ddf59b5df8b] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=71.4 K 2023-03-19 09:55:58,312 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d7f8b1f873974c4fb1e309c38775375c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679219753950 2023-03-19 09:55:58,312 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 773d22000f894dd688b94505b413119a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679219754063 2023-03-19 09:55:58,312 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5d9beae549704ba7b91eb1168d6ddd5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679219755155 2023-03-19 09:55:58,312 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8f610fdd4ea344109b948cbaab72b2d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679219755534 2023-03-19 09:55:58,313 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 34cd6be951dd4571a569d3f6b6757e37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679219756685 2023-03-19 09:55:58,313 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6e605770e2ca453ba8d04ddf59b5df8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679219757830 2023-03-19 09:55:58,322 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#461 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:55:58,357 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:58,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:58,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:58,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:58,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:58,360 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:58,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:58,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:58,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:58,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:58,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:58,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:58,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:58,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:58,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:58,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:58,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:58,365 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-19 09:55:58,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:58,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:58,366 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:58,366 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:58,366 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:58,366 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:55:58,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:58,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:58,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219818581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219818584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219818585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219818686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219818693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219818694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,709 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/c7ff2cb9a96b4c49a3a906076b487b04 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/c7ff2cb9a96b4c49a3a906076b487b04 2023-03-19 09:55:58,714 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into c7ff2cb9a96b4c49a3a906076b487b04(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:58,714 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:58,714 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=10, startTime=1679219758283; duration=0sec 2023-03-19 09:55:58,714 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:58,730 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/bd4e12090dea4cacbb9dde033d9abf74 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd4e12090dea4cacbb9dde033d9abf74 2023-03-19 09:55:58,735 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into bd4e12090dea4cacbb9dde033d9abf74(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:55:58,735 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:58,735 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=10, startTime=1679219758283; duration=0sec 2023-03-19 09:55:58,735 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:55:58,773 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/706f779929dc41aaa058f6ffe0af1146 2023-03-19 09:55:58,783 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/4900e7f461e842f19c02441b60310d19 2023-03-19 09:55:58,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219818894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219818901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:58,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:58,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219818902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,192 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/a26ee9ca865c4f498a5efb983d08211d 2023-03-19 09:55:59,196 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/706f779929dc41aaa058f6ffe0af1146 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/706f779929dc41aaa058f6ffe0af1146 2023-03-19 09:55:59,199 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/706f779929dc41aaa058f6ffe0af1146, entries=150, sequenceid=321, filesize=11.9 K 2023-03-19 09:55:59,200 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/4900e7f461e842f19c02441b60310d19 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4900e7f461e842f19c02441b60310d19 2023-03-19 09:55:59,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:59,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219819197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,203 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4900e7f461e842f19c02441b60310d19, entries=150, sequenceid=321, filesize=11.9 K 2023-03-19 09:55:59,204 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/a26ee9ca865c4f498a5efb983d08211d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a26ee9ca865c4f498a5efb983d08211d 2023-03-19 09:55:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219819206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,208 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a26ee9ca865c4f498a5efb983d08211d, entries=150, sequenceid=321, filesize=11.9 K 2023-03-19 09:55:59,209 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for ddd80e01130df7b3867030aa3a103731 in 844ms, sequenceid=321, compaction requested=false 2023-03-19 09:55:59,209 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:55:59,209 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:59,211 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:55:59,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:55:59,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:59,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:55:59,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:59,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:55:59,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263d858d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:59,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:59,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:59,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:59,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:59,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,225 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/3ab590c2d0a94055810d2c527f1ef118 2023-03-19 09:55:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:59,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:59,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:59,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:59,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:59,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:59,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:59,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:59,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:59,236 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:59,236 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:59,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219759273, max timestamp=1679219759302Num uniques 859; 2023-03-19 09:55:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219759273, max timestamp=1679219759302Num uniques 859; 2023-03-19 09:55:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219759273, max timestamp=1679219759302Num uniques 859; 2023-03-19 09:55:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:55:59,336 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:59,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:59,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:59,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:59,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:59,341 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:59,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:59,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:59,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:59,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:59,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:59,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:59,348 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,348 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:59,348 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:59,348 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:59,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,351 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:59,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d46ebbf[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:59,352 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:59,352 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:59,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,355 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:59,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:59,357 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:59,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:59,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:59,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:59,357 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:59,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,360 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,364 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,364 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:59,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:59,364 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,364 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,364 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,364 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,364 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:59,365 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:59,364 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219819378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,465 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:59,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:59,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:59,468 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:59,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:59,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:59,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,470 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:59,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:59,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:59,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:59,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:59,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:59,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:59,471 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,471 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:59,471 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:59,471 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:59,473 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:59,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@787a47eb[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:59,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:59,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,476 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:59,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:59,478 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:59,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:59,481 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:59,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:59,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:59,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:59,481 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:59,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,484 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:59,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:59,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,487 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,487 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:59,487 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:59,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:59,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219819485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,587 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:59,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:59,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:59,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:59,590 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:59,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:59,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:59,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,591 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,592 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:59,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:59,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:59,594 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,594 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:59,594 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:59,594 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:59,598 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:59,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:59,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca719ca[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:59,601 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,602 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:59,602 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,602 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:59,603 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:59,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:59,606 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:59,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:59,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:59,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:59,606 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:59,607 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,607 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,607 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,608 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,609 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,614 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:59,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:59,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:59,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,615 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:59,615 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:59,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/e364c395a1104bbfa58a875aa945e416 2023-03-19 09:55:59,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:59,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219819691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:59,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219819707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:55:59,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219819709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,715 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:59,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:59,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:59,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:59,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:59,718 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:59,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:59,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:59,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,719 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,720 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,720 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:59,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:59,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:59,721 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,721 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:59,721 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:59,721 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:59,722 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:59,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fcf63e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:59,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:59,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,726 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:59,726 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,726 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:59,729 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:59,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:59,733 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:59,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:59,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:59,733 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:59,733 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:59,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,735 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,735 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,738 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,741 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:59,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,741 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,741 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:59,741 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:59,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:59,842 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:59,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:59,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:59,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:59,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:59,845 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:59,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:59,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:59,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,845 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,847 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:59,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:59,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:59,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:59,848 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,848 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:59,848 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:59,848 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,849 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:59,849 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:59,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:59,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7de44d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:59,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:59,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,854 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:59,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:59,856 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,856 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,856 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,856 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:59,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:59,858 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:59,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:59,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:59,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:59,858 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:59,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,865 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,865 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,865 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:59,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:59,865 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,865 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,866 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:59,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,866 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:59,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,866 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,966 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:55:59,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:55:59,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:55:59,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:55:59,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:55:59,969 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:55:59,969 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,969 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:55:59,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:55:59,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,970 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:55:59,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:55:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:55:59,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:55:59,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:55:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:55:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:55:59,973 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,973 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:55:59,973 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:55:59,973 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:55:59,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:55:59,975 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,975 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,975 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,975 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:55:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:55:59,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6612b736[Count = 0] remaining members to acquire global barrier 2023-03-19 09:55:59,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:55:59,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,978 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:55:59,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:55:59,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,980 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:55:59,981 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,981 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:55:59,983 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:55:59,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:55:59,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:55:59,983 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:55:59,983 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:55:59,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:55:59,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:55:59,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:55:59,985 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:55:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:55:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:55:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:55:59,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:55:59,991 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:55:59,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:55:59,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:55:59,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:55:59,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:55:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:55:59,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,992 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:55:59,992 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:55:59,992 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,992 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:55:59,992 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:55:59,992 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:55:59,992 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:55:59,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:55:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:00,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219819999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/d2d4ade747484c8cb0bcbc7996894087 2023-03-19 09:56:00,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/3ab590c2d0a94055810d2c527f1ef118 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ab590c2d0a94055810d2c527f1ef118 2023-03-19 09:56:00,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ab590c2d0a94055810d2c527f1ef118, entries=100, sequenceid=334, filesize=9.5 K 2023-03-19 09:56:00,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/e364c395a1104bbfa58a875aa945e416 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e364c395a1104bbfa58a875aa945e416 2023-03-19 09:56:00,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e364c395a1104bbfa58a875aa945e416, entries=100, sequenceid=334, filesize=9.5 K 2023-03-19 09:56:00,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/d2d4ade747484c8cb0bcbc7996894087 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d2d4ade747484c8cb0bcbc7996894087 2023-03-19 09:56:00,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d2d4ade747484c8cb0bcbc7996894087, entries=100, sequenceid=334, filesize=9.5 K 2023-03-19 09:56:00,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ddd80e01130df7b3867030aa3a103731 in 876ms, sequenceid=334, compaction requested=true 2023-03-19 09:56:00,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:00,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:00,087 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:00,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:00,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:00,088 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:00,088 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:00,088 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:56:00,088 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:56:00,088 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,088 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,088 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/c7ff2cb9a96b4c49a3a906076b487b04, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/706f779929dc41aaa058f6ffe0af1146, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ab590c2d0a94055810d2c527f1ef118] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=33.9 K 2023-03-19 09:56:00,088 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d1126312d8ad45eda4b05b75068cf4ee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4900e7f461e842f19c02441b60310d19, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e364c395a1104bbfa58a875aa945e416] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=33.9 K 2023-03-19 09:56:00,089 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c7ff2cb9a96b4c49a3a906076b487b04, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679219757830 2023-03-19 09:56:00,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d1126312d8ad45eda4b05b75068cf4ee, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679219757830 2023-03-19 09:56:00,089 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 706f779929dc41aaa058f6ffe0af1146, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679219757925 2023-03-19 09:56:00,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4900e7f461e842f19c02441b60310d19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679219757925 2023-03-19 09:56:00,089 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3ab590c2d0a94055810d2c527f1ef118, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679219758584 2023-03-19 09:56:00,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e364c395a1104bbfa58a875aa945e416, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679219758584 2023-03-19 09:56:00,092 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:00,094 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:00,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:00,097 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#469 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:00,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:00,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:00,097 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:00,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:00,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:00,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:00,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:00,123 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/6bef35923f784e7cba62e9695b429b73 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6bef35923f784e7cba62e9695b429b73 2023-03-19 09:56:00,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:00,128 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:00,129 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:00,129 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:00,138 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into 6bef35923f784e7cba62e9695b429b73(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:00,139 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,139 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=13, startTime=1679219760087; duration=0sec 2023-03-19 09:56:00,139 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:00,139 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:00,141 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:00,141 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:56:00,141 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,141 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd4e12090dea4cacbb9dde033d9abf74, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a26ee9ca865c4f498a5efb983d08211d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d2d4ade747484c8cb0bcbc7996894087] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=33.9 K 2023-03-19 09:56:00,142 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting bd4e12090dea4cacbb9dde033d9abf74, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679219757830 2023-03-19 09:56:00,143 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a26ee9ca865c4f498a5efb983d08211d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679219757925 2023-03-19 09:56:00,143 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d2d4ade747484c8cb0bcbc7996894087, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679219758584 2023-03-19 09:56:00,149 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/63432edb60a54a1eb5ee97d1b3c2349b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/63432edb60a54a1eb5ee97d1b3c2349b 2023-03-19 09:56:00,152 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into 63432edb60a54a1eb5ee97d1b3c2349b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:00,152 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,153 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=13, startTime=1679219760087; duration=0sec 2023-03-19 09:56:00,153 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:00,156 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:00,162 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/39667c1c886747e4bbabb4af4c31bc91 2023-03-19 09:56:00,197 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/de8a29166e08433e99addead96ec92ad 2023-03-19 09:56:00,256 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/47f4a7298ce74992b504b7019d88be18 2023-03-19 09:56:00,261 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/39667c1c886747e4bbabb4af4c31bc91 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39667c1c886747e4bbabb4af4c31bc91 2023-03-19 09:56:00,274 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39667c1c886747e4bbabb4af4c31bc91, entries=150, sequenceid=370, filesize=11.9 K 2023-03-19 09:56:00,274 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/de8a29166e08433e99addead96ec92ad as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/de8a29166e08433e99addead96ec92ad 2023-03-19 09:56:00,279 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/de8a29166e08433e99addead96ec92ad, entries=150, sequenceid=370, filesize=11.9 K 2023-03-19 09:56:00,279 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/47f4a7298ce74992b504b7019d88be18 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/47f4a7298ce74992b504b7019d88be18 2023-03-19 09:56:00,283 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/47f4a7298ce74992b504b7019d88be18, entries=150, sequenceid=370, filesize=11.9 K 2023-03-19 09:56:00,283 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for ddd80e01130df7b3867030aa3a103731 in 154ms, sequenceid=370, compaction requested=false 2023-03-19 09:56:00,283 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,283 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,286 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42df9b77[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:00,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:00,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:00,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:00,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:00,294 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:00,294 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:00,294 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:00,294 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:00,294 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:00,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,303 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:00,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,303 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:00,303 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:00,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,403 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:00,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:00,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:00,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:00,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:00,407 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:00,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:00,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:00,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,407 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,408 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:00,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:00,410 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:00,411 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,411 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:00,411 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,411 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:00,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:00,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:00,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:00,413 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a3c814[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:00,415 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:00,415 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,417 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:00,417 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:00,419 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:00,422 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:00,422 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:00,422 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:00,422 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:00,422 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:00,424 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,424 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,424 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:00,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,428 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,429 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,428 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:00,429 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:00,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:00,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:00,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:00,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:00,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/0ee0eacb9b484da8bf8d04d6914486f7 2023-03-19 09:56:00,529 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:00,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:00,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:00,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:00,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:00,532 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:00,532 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,532 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:00,532 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:00,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,535 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,535 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,536 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,536 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:00,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:00,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/cffe891bc3f44b7e9352258ecc89d352 2023-03-19 09:56:00,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:00,557 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:00,557 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:00,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,559 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@505f05a1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:00,562 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:00,562 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,563 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:00,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:00,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:00,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,565 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:00,565 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,570 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/77ab796f8812400fbe43d1e283d81688 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/77ab796f8812400fbe43d1e283d81688 2023-03-19 09:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:00,572 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:00,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:00,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:00,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:00,572 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:00,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,577 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,578 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into 77ab796f8812400fbe43d1e283d81688(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:00,578 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,578 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=13, startTime=1679219760087; duration=0sec 2023-03-19 09:56:00,578 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:00,579 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:00,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,583 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,583 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:00,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:00,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,583 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5269437854b64a3cbeb9ef0f05dd0dfa 2023-03-19 09:56:00,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/0ee0eacb9b484da8bf8d04d6914486f7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0ee0eacb9b484da8bf8d04d6914486f7 2023-03-19 09:56:00,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0ee0eacb9b484da8bf8d04d6914486f7, entries=200, sequenceid=383, filesize=14.3 K 2023-03-19 09:56:00,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/cffe891bc3f44b7e9352258ecc89d352 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/cffe891bc3f44b7e9352258ecc89d352 2023-03-19 09:56:00,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/cffe891bc3f44b7e9352258ecc89d352, entries=150, sequenceid=383, filesize=11.9 K 2023-03-19 09:56:00,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5269437854b64a3cbeb9ef0f05dd0dfa as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5269437854b64a3cbeb9ef0f05dd0dfa 2023-03-19 09:56:00,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5269437854b64a3cbeb9ef0f05dd0dfa, entries=150, sequenceid=383, filesize=11.9 K 2023-03-19 09:56:00,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=93.93 KB/96180 for ddd80e01130df7b3867030aa3a103731 in 105ms, sequenceid=383, compaction requested=true 2023-03-19 09:56:00,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:00,624 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:00,624 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:00,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:00,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:00,625 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:00,625 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:56:00,626 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,626 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/63432edb60a54a1eb5ee97d1b3c2349b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/de8a29166e08433e99addead96ec92ad, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/cffe891bc3f44b7e9352258ecc89d352] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=36.4 K 2023-03-19 09:56:00,626 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:00,626 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:56:00,626 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,626 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6bef35923f784e7cba62e9695b429b73, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39667c1c886747e4bbabb4af4c31bc91, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0ee0eacb9b484da8bf8d04d6914486f7] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=38.8 K 2023-03-19 09:56:00,627 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 6bef35923f784e7cba62e9695b429b73, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679219757935 2023-03-19 09:56:00,627 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 63432edb60a54a1eb5ee97d1b3c2349b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679219757935 2023-03-19 09:56:00,628 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 39667c1c886747e4bbabb4af4c31bc91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1679219759369 2023-03-19 09:56:00,628 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0ee0eacb9b484da8bf8d04d6914486f7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679219760507 2023-03-19 09:56:00,628 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting de8a29166e08433e99addead96ec92ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1679219759369 2023-03-19 09:56:00,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-19 09:56:00,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:00,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:00,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:00,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:00,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:00,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:00,629 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting cffe891bc3f44b7e9352258ecc89d352, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679219760507 2023-03-19 09:56:00,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:00,649 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#478 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:00,653 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#479 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:00,678 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/55b6914fdb7a4e5c990b5ddd32834c6b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/55b6914fdb7a4e5c990b5ddd32834c6b 2023-03-19 09:56:00,683 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into 55b6914fdb7a4e5c990b5ddd32834c6b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:00,683 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,683 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=13, startTime=1679219760624; duration=0sec 2023-03-19 09:56:00,683 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:00,683 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:00,684 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:00,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:00,685 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:00,685 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:56:00,685 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,685 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/77ab796f8812400fbe43d1e283d81688, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/47f4a7298ce74992b504b7019d88be18, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5269437854b64a3cbeb9ef0f05dd0dfa] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=36.4 K 2023-03-19 09:56:00,685 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 77ab796f8812400fbe43d1e283d81688, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679219757935 2023-03-19 09:56:00,686 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 47f4a7298ce74992b504b7019d88be18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1679219759369 2023-03-19 09:56:00,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:00,686 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5269437854b64a3cbeb9ef0f05dd0dfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679219760507 2023-03-19 09:56:00,691 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/0d4475b3f0644f699ce390c6a9db2daf as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0d4475b3f0644f699ce390c6a9db2daf 2023-03-19 09:56:00,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:00,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:00,692 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:00,692 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,692 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:00,692 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:00,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,693 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,696 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,696 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into 0d4475b3f0644f699ce390c6a9db2daf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:00,697 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:00,697 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=13, startTime=1679219760624; duration=0sec 2023-03-19 09:56:00,697 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,697 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,697 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:00,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:00,701 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:00,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:00,704 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,704 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:00,704 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:00,704 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:00,707 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,707 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,707 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,707 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2307a39b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:00,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:00,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,710 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:00,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:00,711 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:00,712 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,712 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:00,712 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:00,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:00,715 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:00,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:00,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:00,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:00,715 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:00,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:00,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219820712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,723 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:00,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:00,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,723 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:00,723 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,723 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:00,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:00,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219820721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:00,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219820722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,823 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:00,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:00,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:00,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:00,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:00,829 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:00,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:00,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:00,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:00,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219820823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,831 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:00,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:00,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:00,851 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,851 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:00,851 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:00,851 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:00,857 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3d90af[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:00,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:00,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,862 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:00,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,862 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:00,864 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:00,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:00,867 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:00,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:00,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:00,867 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:00,867 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:00,868 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,868 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,868 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,873 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,876 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:00,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:00,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:00,877 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:00,877 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:00,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,877 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:00,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:00,977 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:00,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:00,982 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:00,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:00,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:00,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:00,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:00,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:00,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:00,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:00,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:00,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:00,988 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:00,988 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,989 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:00,989 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:00,989 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:00,991 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:00,991 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,992 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:00,992 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:00,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70330874[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:00,995 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:00,995 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,996 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:00,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:00,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:00,996 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:00,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:00,999 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:00,999 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:00,999 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:00,999 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:01,003 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:01,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:01,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:01,003 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:01,003 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:01,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,005 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,005 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:01,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:01,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,012 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,012 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:01,012 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:01,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:01,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219821032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/fe3db23c0d5145bcb09614fca0ddfc7a 2023-03-19 09:56:01,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/ac9506d851194b37b197653efb831c9c 2023-03-19 09:56:01,113 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:01,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:01,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:01,116 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5247d93b303a40d59f8562bead17e1fe as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5247d93b303a40d59f8562bead17e1fe 2023-03-19 09:56:01,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:01,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:01,117 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:01,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:01,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:01,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,119 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,124 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,125 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,125 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:01,127 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into 5247d93b303a40d59f8562bead17e1fe(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:01,127 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:01,127 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=13, startTime=1679219760624; duration=0sec 2023-03-19 09:56:01,127 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:01,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:01,135 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:01,135 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,135 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:01,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:01,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:01,136 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,136 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:01,136 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:01,136 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:01,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:01,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:01,136 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:01,138 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,138 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,138 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,138 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e979de[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:01,140 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:01,140 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,148 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:01,150 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:01,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:01,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:01,152 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:01,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:01,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:01,152 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:01,152 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:01,154 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,154 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,154 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,166 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,166 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,166 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:01,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:01,166 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,166 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:01,166 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,167 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:01,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:01,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,267 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:01,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:01,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:01,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:01,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:01,270 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:01,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:01,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:01,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,273 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:01,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:01,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:01,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:01,275 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:01,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:01,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:01,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:01,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:01,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:01,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466c0223[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:01,279 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:01,279 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:01,280 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:01,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:01,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:01,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:01,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:01,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:01,290 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:01,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:01,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:01,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:01,290 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:01,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,293 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,293 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,297 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:01,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:01,297 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:01,298 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:01,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:01,298 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:01,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219821335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,398 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:01,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:01,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:01,416 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:01,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:01,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:01,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,417 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,418 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,418 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:01,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:01,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:01,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:01,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:01,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:01,437 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:01,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,439 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:01,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a55aa9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:01,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:01,442 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,447 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,447 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:01,447 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:01,447 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:01,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:01,450 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,450 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:01,450 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,450 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:01,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:01,453 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:01,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:01,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:01,453 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:01,453 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:01,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,455 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,461 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,462 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:01,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:01,463 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:01,463 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:01,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/4e726915bac74526800ab67fb228c7a8 2023-03-19 09:56:01,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/fe3db23c0d5145bcb09614fca0ddfc7a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/fe3db23c0d5145bcb09614fca0ddfc7a 2023-03-19 09:56:01,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/fe3db23c0d5145bcb09614fca0ddfc7a, entries=200, sequenceid=402, filesize=14.3 K 2023-03-19 09:56:01,552 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/ac9506d851194b37b197653efb831c9c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/ac9506d851194b37b197653efb831c9c 2023-03-19 09:56:01,563 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:01,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:01,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:01,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:01,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:01,572 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:01,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:01,572 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:01,573 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,573 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/ac9506d851194b37b197653efb831c9c, entries=150, sequenceid=402, filesize=11.9 K 2023-03-19 09:56:01,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/4e726915bac74526800ab67fb228c7a8 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4e726915bac74526800ab67fb228c7a8 2023-03-19 09:56:01,579 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,580 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:01,584 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:01,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4e726915bac74526800ab67fb228c7a8, entries=150, sequenceid=402, filesize=11.9 K 2023-03-19 09:56:01,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ddd80e01130df7b3867030aa3a103731 in 966ms, sequenceid=402, compaction requested=false 2023-03-19 09:56:01,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:01,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:01,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:01,600 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,600 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:01,600 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-19 09:56:01,601 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:01,601 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:01,601 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:01,601 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:01,601 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:01,601 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:01,644 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/f45ef1882a7b4d1191b1379b4993d7ba 2023-03-19 09:56:01,676 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d 2023-03-19 09:56:01,707 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/8948cdfdef3648bdaac3b9b6f2d00ba1 2023-03-19 09:56:01,713 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/f45ef1882a7b4d1191b1379b4993d7ba as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f45ef1882a7b4d1191b1379b4993d7ba 2023-03-19 09:56:01,718 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f45ef1882a7b4d1191b1379b4993d7ba, entries=150, sequenceid=419, filesize=11.9 K 2023-03-19 09:56:01,719 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d 2023-03-19 09:56:01,725 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d, entries=150, sequenceid=419, filesize=11.9 K 2023-03-19 09:56:01,726 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/8948cdfdef3648bdaac3b9b6f2d00ba1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8948cdfdef3648bdaac3b9b6f2d00ba1 2023-03-19 09:56:01,739 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8948cdfdef3648bdaac3b9b6f2d00ba1, entries=150, sequenceid=419, filesize=11.9 K 2023-03-19 09:56:01,740 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for ddd80e01130df7b3867030aa3a103731 in 140ms, sequenceid=419, compaction requested=true 2023-03-19 09:56:01,740 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:01,740 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,740 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:01,740 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:01,740 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:01,744 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:01,746 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:01,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f3d8481[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:01,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:01,749 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,751 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:01,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:01,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:01,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,752 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:01,764 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:01,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:01,772 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:01,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:01,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:01,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:01,772 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:01,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,774 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,774 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:01,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:01,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,780 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,780 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:01,780 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,781 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:01,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:56:01,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:01,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:01,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:01,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:01,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:01,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:01,880 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:01,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:01,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:01,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:01,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:01,884 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:01,884 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,884 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:01,884 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:01,884 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,884 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,886 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:01,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:01,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:01,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:01,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:01,888 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:01,892 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,892 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:01,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/a532d13bcb9f4cb4aac0f080ba3a8ac1 2023-03-19 09:56:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c893131[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:01,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:01,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:01,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:01,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:01,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:01,901 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:01,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:01,904 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:01,904 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:01,904 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:01,904 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:01,904 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:01,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:01,906 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:01,907 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:01,912 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:01,912 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:01,912 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,912 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:01,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:01,912 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:01,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:01,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:01,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:01,912 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:01,913 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:01,913 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:01,913 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:01,913 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,913 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:01,913 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:01,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:01,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:01,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/2cc44c8b950146f386b7c4c39cb17b0d 2023-03-19 09:56:02,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219761979, max timestamp=1679219762003Num uniques 859; 2023-03-19 09:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219761979, max timestamp=1679219762003Num uniques 859; 2023-03-19 09:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:02,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219761979, max timestamp=1679219762003Num uniques 859; 2023-03-19 09:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:02,013 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:02,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:02,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:02,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:02,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:02,016 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:02,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:02,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:02,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,017 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,017 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,018 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,018 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:02,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:02,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:02,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:02,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:02,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:02,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:02,019 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,020 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:02,020 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:02,020 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:02,021 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:02,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad25f99[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:02,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:02,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,024 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,025 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:02,029 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:02,031 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,031 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:02,031 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,032 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:02,034 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:02,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:02,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:02,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:02,034 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:02,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,036 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,037 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,039 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,039 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,040 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,039 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:02,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:02,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,040 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:02,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,040 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:02,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219822121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,140 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:02,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:02,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:02,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:02,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:02,144 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:02,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:02,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:02,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,144 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,145 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,145 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,146 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:02,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:02,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:02,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:02,149 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,149 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,150 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:02,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77bf3078[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:02,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:02,151 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,152 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:02,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:02,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,154 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:02,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:02,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:02,154 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:02,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:02,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:02,154 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:02,154 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:02,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,156 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,156 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:02,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,164 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,164 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,164 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,164 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219822223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,264 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:02,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:02,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:02,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48224 deadline: 1679219822261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:02,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:02,267 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:02,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:02,269 DEBUG [Thread-3192] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:02,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:02,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,269 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,270 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,270 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:02,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:02,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:02,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:02,272 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:02,274 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:02,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4495ab82[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:02,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:02,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:02,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:02,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:02,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:02,279 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:02,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,279 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:02,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:02,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:02,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:02,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:02,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:02,281 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:02,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,283 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,284 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,286 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48228 deadline: 1679219822279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,289 DEBUG [Thread-3188] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8205 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:02,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,289 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:02,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:02,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:02,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,290 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:02,290 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:02,348 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/cd1c066948b74761a441e1e4f7e7e274 2023-03-19 09:56:02,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/a532d13bcb9f4cb4aac0f080ba3a8ac1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a532d13bcb9f4cb4aac0f080ba3a8ac1 2023-03-19 09:56:02,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a532d13bcb9f4cb4aac0f080ba3a8ac1, entries=200, sequenceid=429, filesize=14.3 K 2023-03-19 09:56:02,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/2cc44c8b950146f386b7c4c39cb17b0d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2cc44c8b950146f386b7c4c39cb17b0d 2023-03-19 09:56:02,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2cc44c8b950146f386b7c4c39cb17b0d, entries=150, sequenceid=429, filesize=11.9 K 2023-03-19 09:56:02,364 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/cd1c066948b74761a441e1e4f7e7e274 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/cd1c066948b74761a441e1e4f7e7e274 2023-03-19 09:56:02,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/cd1c066948b74761a441e1e4f7e7e274, entries=150, sequenceid=429, filesize=11.9 K 2023-03-19 09:56:02,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ddd80e01130df7b3867030aa3a103731 in 500ms, sequenceid=429, compaction requested=true 2023-03-19 09:56:02,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:02,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:02,370 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:02,370 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:02,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:02,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:02,371 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54476 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:02,372 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:02,372 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:56:02,372 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:56:02,372 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,372 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,372 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/55b6914fdb7a4e5c990b5ddd32834c6b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/fe3db23c0d5145bcb09614fca0ddfc7a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f45ef1882a7b4d1191b1379b4993d7ba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a532d13bcb9f4cb4aac0f080ba3a8ac1] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=53.2 K 2023-03-19 09:56:02,372 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0d4475b3f0644f699ce390c6a9db2daf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/ac9506d851194b37b197653efb831c9c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2cc44c8b950146f386b7c4c39cb17b0d] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=48.4 K 2023-03-19 09:56:02,373 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0d4475b3f0644f699ce390c6a9db2daf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679219760507 2023-03-19 09:56:02,373 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 55b6914fdb7a4e5c990b5ddd32834c6b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679219760507 2023-03-19 09:56:02,374 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ac9506d851194b37b197653efb831c9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679219760616 2023-03-19 09:56:02,374 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting fe3db23c0d5145bcb09614fca0ddfc7a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679219760607 2023-03-19 09:56:02,374 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0ccf4a0929ea4bdf8c07ec27e8dd5a2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679219760669 2023-03-19 09:56:02,374 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting f45ef1882a7b4d1191b1379b4993d7ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679219760669 2023-03-19 09:56:02,374 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2cc44c8b950146f386b7c4c39cb17b0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679219761855 2023-03-19 09:56:02,374 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting a532d13bcb9f4cb4aac0f080ba3a8ac1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679219761845 2023-03-19 09:56:02,383 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:02,384 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#490 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:02,390 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:02,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:02,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:02,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:02,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:02,398 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:02,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:02,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:02,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,398 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,400 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:02,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:02,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:02,417 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:02,429 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/6aa08c80c43b46f1992283b28246f883 2023-03-19 09:56:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:02,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:02,445 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/965a6c6fb80a476385bb6c31479279ab 2023-03-19 09:56:02,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219822485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219822586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219822746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219822746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,753 DEBUG [Thread-3196] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:02,753 DEBUG [Thread-3190] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:02,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:02,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219822797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/e266e83259da45f391a237921572a2b6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e266e83259da45f391a237921572a2b6 2023-03-19 09:56:02,813 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/83b16148cd804b4f95345132d423d582 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/83b16148cd804b4f95345132d423d582 2023-03-19 09:56:02,815 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into e266e83259da45f391a237921572a2b6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:02,815 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:02,815 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=12, startTime=1679219762370; duration=0sec 2023-03-19 09:56:02,815 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:02,815 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:02,816 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:02,817 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:56:02,817 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,817 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into 83b16148cd804b4f95345132d423d582(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:02,817 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5247d93b303a40d59f8562bead17e1fe, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4e726915bac74526800ab67fb228c7a8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8948cdfdef3648bdaac3b9b6f2d00ba1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/cd1c066948b74761a441e1e4f7e7e274] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=48.4 K 2023-03-19 09:56:02,817 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:02,817 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=12, startTime=1679219762370; duration=0sec 2023-03-19 09:56:02,817 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:02,817 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5247d93b303a40d59f8562bead17e1fe, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679219760507 2023-03-19 09:56:02,817 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 4e726915bac74526800ab67fb228c7a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679219760616 2023-03-19 09:56:02,818 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 8948cdfdef3648bdaac3b9b6f2d00ba1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679219760669 2023-03-19 09:56:02,818 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting cd1c066948b74761a441e1e4f7e7e274, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679219761855 2023-03-19 09:56:02,823 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:02,830 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5bf575fb303d4ab080e95dc6827f1c6a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5bf575fb303d4ab080e95dc6827f1c6a 2023-03-19 09:56:02,834 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into 5bf575fb303d4ab080e95dc6827f1c6a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:02,834 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:02,834 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=12, startTime=1679219762370; duration=0sec 2023-03-19 09:56:02,834 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:02,861 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/d94c866f4e3f462993bc51044c6b6c16 2023-03-19 09:56:02,864 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/6aa08c80c43b46f1992283b28246f883 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6aa08c80c43b46f1992283b28246f883 2023-03-19 09:56:02,867 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6aa08c80c43b46f1992283b28246f883, entries=150, sequenceid=466, filesize=11.9 K 2023-03-19 09:56:02,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/965a6c6fb80a476385bb6c31479279ab as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/965a6c6fb80a476385bb6c31479279ab 2023-03-19 09:56:02,871 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/965a6c6fb80a476385bb6c31479279ab, entries=150, sequenceid=466, filesize=11.9 K 2023-03-19 09:56:02,871 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/d94c866f4e3f462993bc51044c6b6c16 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d94c866f4e3f462993bc51044c6b6c16 2023-03-19 09:56:02,874 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d94c866f4e3f462993bc51044c6b6c16, entries=150, sequenceid=466, filesize=11.9 K 2023-03-19 09:56:02,875 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for ddd80e01130df7b3867030aa3a103731 in 458ms, sequenceid=466, compaction requested=false 2023-03-19 09:56:02,875 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:02,875 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:02,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:02,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41901a98[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:02,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:02,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,879 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:02,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:02,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:02,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,879 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:02,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,880 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:02,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:02,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:02,883 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:02,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:02,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:02,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:02,883 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:02,885 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,885 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:02,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,885 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:02,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,889 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:02,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:02,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:02,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:02,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:02,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:02,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,890 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:02,890 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:02,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:02,890 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,990 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:02,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:02,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:02,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:02,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:02,993 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:02,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:02,993 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:02,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,994 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,995 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:02,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:02,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:02,996 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:02,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:02,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:02,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:02,997 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:02,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:02,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:02,997 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:56:02,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:02,997 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:02,998 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:02,998 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:02,998 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:02,998 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:03,004 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/ef17193e3ad74e6f95a100d338e5ad76 2023-03-19 09:56:03,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:03,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:03,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219763168, max timestamp=1679219763215Num uniques 859; 2023-03-19 09:56:03,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:03,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219763168, max timestamp=1679219763215Num uniques 859; 2023-03-19 09:56:03,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:03,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219763168, max timestamp=1679219763215Num uniques 859; 2023-03-19 09:56:03,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:03,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:03,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219823316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,413 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/d01d2317bedf4c31b291050defd024f4 2023-03-19 09:56:03,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:03,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219823426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:03,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219823629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,824 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/dfe3d9ccd92c4968a07d38316abb787d 2023-03-19 09:56:03,827 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/ef17193e3ad74e6f95a100d338e5ad76 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/ef17193e3ad74e6f95a100d338e5ad76 2023-03-19 09:56:03,830 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/ef17193e3ad74e6f95a100d338e5ad76, entries=150, sequenceid=479, filesize=11.9 K 2023-03-19 09:56:03,831 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/d01d2317bedf4c31b291050defd024f4 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d01d2317bedf4c31b291050defd024f4 2023-03-19 09:56:03,834 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d01d2317bedf4c31b291050defd024f4, entries=150, sequenceid=479, filesize=11.9 K 2023-03-19 09:56:03,835 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/dfe3d9ccd92c4968a07d38316abb787d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/dfe3d9ccd92c4968a07d38316abb787d 2023-03-19 09:56:03,838 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/dfe3d9ccd92c4968a07d38316abb787d, entries=150, sequenceid=479, filesize=11.9 K 2023-03-19 09:56:03,839 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ddd80e01130df7b3867030aa3a103731 in 842ms, sequenceid=479, compaction requested=true 2023-03-19 09:56:03,839 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:03,839 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:03,839 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:03,841 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:03,842 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,842 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3946aba4[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:03,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:03,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,848 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:03,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:03,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:03,849 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:03,852 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:03,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:03,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:03,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:03,852 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:03,854 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:03,854 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,854 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:03,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:03,854 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:03,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:03,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:03,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:03,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:03,861 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:03,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:03,862 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:03,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:03,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:03,862 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,862 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,862 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,862 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:03,862 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:03,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:03,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-19 09:56:03,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:03,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:56:03,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:03,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:56:03,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:03,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:56:03,962 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:03,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:03,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:03,965 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:03,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:03,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:03,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:03,966 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,966 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:03,967 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:03,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:03,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:03,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:03,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:03,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:03,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:03,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:03,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:03,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:03,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:03,969 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:03,969 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:03,969 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:03,969 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:03,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:03,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:03,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:03,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:03,970 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:03,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:03,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,971 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:03,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:03,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:03,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:03,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:03,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f47004[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:03,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:03,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,973 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:03,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:03,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:03,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:03,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:03,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:03,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,976 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:03,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:03,978 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:03,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:03,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:03,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:03,978 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:03,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:03,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:03,981 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,982 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:03,984 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:03,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:03,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:03,984 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:03,984 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:03,984 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:03,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:03,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219823988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,084 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:04,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:04,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:04,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:04,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:04,088 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:04,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:04,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:04,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,088 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,090 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:04,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:04,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:04,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:04,092 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,092 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:04,092 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:04,092 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:04,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:04,097 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:04,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed404c3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:04,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:04,099 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,101 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,101 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:04,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:04,103 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219824096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,106 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,107 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,107 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:04,107 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:04,107 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:04,107 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:04,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,108 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:04,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:04,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:04,111 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:04,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:04,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:04,111 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:04,111 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:04,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,113 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,114 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:04,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,121 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,121 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,122 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:04,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,122 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:04,222 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:04,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:04,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:04,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:04,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:04,225 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:04,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:04,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:04,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,226 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,226 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,227 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,227 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:04,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:04,227 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:04,228 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,228 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:04,228 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:04,228 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:04,228 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:04,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,229 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,230 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,230 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:04,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f51815b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:04,231 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:04,231 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,232 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:04,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:04,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:04,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:04,235 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,235 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:04,235 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,235 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:04,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:04,237 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:04,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:04,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:04,237 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:04,237 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:04,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,239 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,240 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,242 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,242 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,242 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,242 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,242 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:04,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:04,242 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,243 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:04,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,243 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,243 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:04,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:04,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219824306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,343 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:04,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:04,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:04,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:04,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:04,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/23dd9879bd5c411ba3ed36b99c48ce81 2023-03-19 09:56:04,346 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:04,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:04,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:04,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,347 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,353 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,353 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:04,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:04,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:04,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:04,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:04,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:04,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:04,357 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,357 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:04,357 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:04,357 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:04,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,365 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c6129e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:04,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:04,365 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,366 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:04,366 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:04,366 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:04,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,367 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:04,369 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,369 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:04,369 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,369 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:04,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:04,372 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:04,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:04,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:04,372 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:04,372 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:04,373 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,374 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,374 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,375 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:04,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,377 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,378 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,377 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:04,377 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:04,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,378 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:04,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,478 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:04,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:04,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:04,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:04,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:04,481 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:04,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:04,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:04,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,481 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,483 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:04,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:04,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:04,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:04,484 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,485 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:04,485 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:04,485 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:04,488 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:04,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72624b2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:04,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:04,489 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,490 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,490 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:04,490 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:04,490 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,491 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,492 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:04,495 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:04,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:04,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:04,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:04,495 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:04,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,496 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,497 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,498 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,507 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:04,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:04,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:04,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,507 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:04,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,507 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:04,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,507 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,607 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:04,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:04,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:04,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:04,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:04,611 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:04,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:04,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:04,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,611 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,613 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:04,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:04,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:04,615 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,615 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:04,615 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:04,615 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:04,616 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,616 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:04,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79aaca89[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:04,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:04,619 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:04,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219824614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,621 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:04,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:04,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:04,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:04,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:04,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:04,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:04,633 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:04,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:04,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:04,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:04,633 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:04,635 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,636 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,637 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,640 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,640 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,640 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,641 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:04,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:04,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:04,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:04,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,641 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:04,641 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:04,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,641 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,741 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:04,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:04,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:04,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:04,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:04,745 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,747 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:04,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:04,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:04,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:04,752 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:04,758 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,758 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:04,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/47684f7a549a4cc0ba313a49e795b37f 2023-03-19 09:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:04,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a62c7be[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:04,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:04,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:04,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:04,764 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:04,777 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:04,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:04,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:04,777 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:04,777 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:04,779 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,784 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,787 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:04,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,787 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:04,787 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:04,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,887 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:04,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:04,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:04,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:04,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:04,891 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:04,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:04,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:04,892 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,892 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,893 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,893 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,893 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,894 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:04,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:04,895 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,895 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:04,895 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:04,895 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:04,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49936a8e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:04,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:04,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,899 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:04,901 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:04,903 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:04,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:04,905 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:04,905 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:04,905 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:04,905 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:04,905 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:04,908 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,908 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,909 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:04,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,911 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,913 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,916 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:04,916 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:04,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:04,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:04,916 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,917 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,917 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:04,917 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:04,917 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,917 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:04,917 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:04,916 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:04,917 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:04,916 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:04,917 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:04,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:05,017 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:05,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:05,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:05,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:05,033 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:05,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:05,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:05,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:05,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,038 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:05,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:05,039 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,039 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:05,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:05,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:05,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:05,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:05,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:05,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:05,049 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,050 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:05,050 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:05,050 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:05,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:05,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:05,050 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:05,051 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,051 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:05,052 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,052 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa6d4ba[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:05,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:05,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:05,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:05,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:05,055 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,055 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:05,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:05,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:05,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:05,056 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:05,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:05,059 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:05,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:05,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:05,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:05,059 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:05,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:05,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:05,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:05,062 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,063 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,067 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:05,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:05,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:05,067 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:05,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:05,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:05,067 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:05,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:05,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:05,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:05,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219825125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,168 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:05,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:05,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:05,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:05,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:05,171 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:05,171 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:05,171 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:05,171 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:05,172 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,172 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,174 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,174 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:05,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:05,174 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,174 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:05,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:05,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:05,175 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:05,176 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,176 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:05,176 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:05,176 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:05,176 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:05,178 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5077a745[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:05,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:05,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,184 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,184 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:05,184 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:05,184 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:05,184 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,184 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:05,189 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:05,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:05,191 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:05,191 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:05,191 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:05,191 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:05,191 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:05,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,195 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:05,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:05,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:05,195 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:05,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/88eb6b87763c44ee9bb1433a6530615d 2023-03-19 09:56:05,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,197 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:05,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:05,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:05,207 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:05,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:05,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:05,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,207 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:05,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:05,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:05,207 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:05,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/23dd9879bd5c411ba3ed36b99c48ce81 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/23dd9879bd5c411ba3ed36b99c48ce81 2023-03-19 09:56:05,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,208 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,208 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,208 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,208 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:05,208 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:05,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/23dd9879bd5c411ba3ed36b99c48ce81, entries=200, sequenceid=517, filesize=14.3 K 2023-03-19 09:56:05,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/47684f7a549a4cc0ba313a49e795b37f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/47684f7a549a4cc0ba313a49e795b37f 2023-03-19 09:56:05,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/47684f7a549a4cc0ba313a49e795b37f, entries=150, sequenceid=517, filesize=11.9 K 2023-03-19 09:56:05,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/88eb6b87763c44ee9bb1433a6530615d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/88eb6b87763c44ee9bb1433a6530615d 2023-03-19 09:56:05,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/88eb6b87763c44ee9bb1433a6530615d, entries=150, sequenceid=517, filesize=11.9 K 2023-03-19 09:56:05,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for ddd80e01130df7b3867030aa3a103731 in 1288ms, sequenceid=517, compaction requested=true 2023-03-19 09:56:05,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:05,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:05,226 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:05,226 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:05,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:05,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:05,227 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52172 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:05,227 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:05,227 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:56:05,227 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:56:05,227 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,227 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,227 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e266e83259da45f391a237921572a2b6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6aa08c80c43b46f1992283b28246f883, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/ef17193e3ad74e6f95a100d338e5ad76, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/23dd9879bd5c411ba3ed36b99c48ce81] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=50.9 K 2023-03-19 09:56:05,227 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/83b16148cd804b4f95345132d423d582, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/965a6c6fb80a476385bb6c31479279ab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d01d2317bedf4c31b291050defd024f4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/47684f7a549a4cc0ba313a49e795b37f] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=48.6 K 2023-03-19 09:56:05,228 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting e266e83259da45f391a237921572a2b6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679219761855 2023-03-19 09:56:05,228 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 83b16148cd804b4f95345132d423d582, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679219761855 2023-03-19 09:56:05,228 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 6aa08c80c43b46f1992283b28246f883, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679219762106 2023-03-19 09:56:05,228 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 965a6c6fb80a476385bb6c31479279ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679219762106 2023-03-19 09:56:05,229 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting ef17193e3ad74e6f95a100d338e5ad76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1679219762452 2023-03-19 09:56:05,229 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d01d2317bedf4c31b291050defd024f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1679219762452 2023-03-19 09:56:05,229 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 23dd9879bd5c411ba3ed36b99c48ce81, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679219763272 2023-03-19 09:56:05,229 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 47684f7a549a4cc0ba313a49e795b37f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679219763272 2023-03-19 09:56:05,236 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#501 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:05,238 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#502 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:05,253 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/d5b3c02ac74044a59ea6d38f85b94f7d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d5b3c02ac74044a59ea6d38f85b94f7d 2023-03-19 09:56:05,254 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/03c355234729402a9b5129ddc7e56c7a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03c355234729402a9b5129ddc7e56c7a 2023-03-19 09:56:05,258 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into d5b3c02ac74044a59ea6d38f85b94f7d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:05,258 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:05,258 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=12, startTime=1679219765226; duration=0sec 2023-03-19 09:56:05,258 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:05,259 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:05,260 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:05,260 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:56:05,260 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,260 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5bf575fb303d4ab080e95dc6827f1c6a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d94c866f4e3f462993bc51044c6b6c16, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/dfe3d9ccd92c4968a07d38316abb787d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/88eb6b87763c44ee9bb1433a6530615d] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=48.6 K 2023-03-19 09:56:05,260 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into 03c355234729402a9b5129ddc7e56c7a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:05,260 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:05,261 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=12, startTime=1679219765226; duration=0sec 2023-03-19 09:56:05,261 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:05,263 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5bf575fb303d4ab080e95dc6827f1c6a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679219761855 2023-03-19 09:56:05,263 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d94c866f4e3f462993bc51044c6b6c16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679219762106 2023-03-19 09:56:05,264 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting dfe3d9ccd92c4968a07d38316abb787d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1679219762452 2023-03-19 09:56:05,265 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 88eb6b87763c44ee9bb1433a6530615d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679219763272 2023-03-19 09:56:05,294 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#503 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:05,308 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:05,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:05,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:05,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:05,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:05,315 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:05,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:05,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:05,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:05,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:05,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:05,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:05,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:05,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:05,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:05,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:05,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:05,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:05,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:05,344 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:05,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:05,363 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/d6b0e2b59fe74d5c87df37185f01f7e9 2023-03-19 09:56:05,720 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/5d9bdedca42848c8a339fe10ba51881f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9bdedca42848c8a339fe10ba51881f 2023-03-19 09:56:05,724 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into 5d9bdedca42848c8a339fe10ba51881f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:05,724 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:05,724 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=12, startTime=1679219765226; duration=0sec 2023-03-19 09:56:05,724 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:05,779 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/4e27eaa58bec4426b9559426d3f23a04 2023-03-19 09:56:06,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:06,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:06,189 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/bd6b1f3645044b9aa1d83e4ec80a6a19 2023-03-19 09:56:06,193 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/d6b0e2b59fe74d5c87df37185f01f7e9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d6b0e2b59fe74d5c87df37185f01f7e9 2023-03-19 09:56:06,197 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d6b0e2b59fe74d5c87df37185f01f7e9, entries=150, sequenceid=528, filesize=11.9 K 2023-03-19 09:56:06,197 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/4e27eaa58bec4426b9559426d3f23a04 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e27eaa58bec4426b9559426d3f23a04 2023-03-19 09:56:06,201 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e27eaa58bec4426b9559426d3f23a04, entries=150, sequenceid=528, filesize=11.9 K 2023-03-19 09:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/bd6b1f3645044b9aa1d83e4ec80a6a19 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd6b1f3645044b9aa1d83e4ec80a6a19 2023-03-19 09:56:06,205 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd6b1f3645044b9aa1d83e4ec80a6a19, entries=150, sequenceid=528, filesize=11.9 K 2023-03-19 09:56:06,205 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=87.22 KB/89310 for ddd80e01130df7b3867030aa3a103731 in 861ms, sequenceid=528, compaction requested=false 2023-03-19 09:56:06,205 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:06,205 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:06,206 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:06,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:06,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:56:06,207 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:06,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:06,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:06,207 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:06,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,208 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:06,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:06,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd4c5c2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:06,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:06,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,221 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:06,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:06,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:06,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,222 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:06,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:06,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:06,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:06,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:06,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:06,226 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:06,226 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:06,226 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:06,226 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:06,226 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:06,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,231 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:06,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,233 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,235 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:06,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:06,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:06,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:06,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,238 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:06,239 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:06,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:06,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:06,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/35cb3701dbe6464db5dcd44e20be7a18 2023-03-19 09:56:06,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:06,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219826291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,339 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:06,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:06,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:06,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:06,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:06,341 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:06,341 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:06,341 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:06,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:06,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:06,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:06,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:06,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:06,345 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:06,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:06,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:06,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:06,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:06,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:06,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:06,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:06,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:06,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:06,352 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:06,352 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. started... 2023-03-19 09:56:06,352 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. as already flushing 2023-03-19 09:56:06,352 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:06,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:06,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:06,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:06,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:06,354 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,356 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:06,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec10dcc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:06,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:06,357 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,359 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,359 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:06,359 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:06,359 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:06,359 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,359 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:06,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:06,361 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:06,361 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:06,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:06,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:06,363 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:06,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:06,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:06,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:06,363 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:06,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:06,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,367 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:06,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,368 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:06,369 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:06,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:06,371 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:06,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:06,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:06,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:06,371 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:06,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:06,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:06,372 DEBUG [Thread-3198] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:06,372 DEBUG [Thread-3198] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:06,377 DEBUG [Thread-3205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:54245 2023-03-19 09:56:06,377 DEBUG [Thread-3205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:06,384 DEBUG [Thread-3199] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461f5f3f to 127.0.0.1:54245 2023-03-19 09:56:06,385 DEBUG [Thread-3199] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:06,386 DEBUG [Thread-3207] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:54245 2023-03-19 09:56:06,386 DEBUG [Thread-3207] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:06,387 DEBUG [Thread-3203] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:54245 2023-03-19 09:56:06,387 DEBUG [Thread-3203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:06,394 DEBUG [Thread-3201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:54245 2023-03-19 09:56:06,394 DEBUG [Thread-3201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:06,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:06,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219826400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,472 DEBUG [Thread-3198] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:06,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:06,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:06,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219826603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/03ded2db284243509c574a30db036d1d 2023-03-19 09:56:06,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:06,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48300 deadline: 1679219826757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,759 DEBUG [Thread-3190] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:06,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:06,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48270 deadline: 1679219826774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:06,776 DEBUG [Thread-3196] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:06,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:06,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48226 deadline: 1679219826904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:07,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/d157656231334e6e9ee7618543a2fc67 2023-03-19 09:56:07,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/35cb3701dbe6464db5dcd44e20be7a18 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/35cb3701dbe6464db5dcd44e20be7a18 2023-03-19 09:56:07,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/35cb3701dbe6464db5dcd44e20be7a18, entries=150, sequenceid=546, filesize=11.9 K 2023-03-19 09:56:07,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/03ded2db284243509c574a30db036d1d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03ded2db284243509c574a30db036d1d 2023-03-19 09:56:07,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03ded2db284243509c574a30db036d1d, entries=150, sequenceid=546, filesize=11.9 K 2023-03-19 09:56:07,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/d157656231334e6e9ee7618543a2fc67 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d157656231334e6e9ee7618543a2fc67 2023-03-19 09:56:07,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d157656231334e6e9ee7618543a2fc67, entries=150, sequenceid=546, filesize=11.9 K 2023-03-19 09:56:07,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ddd80e01130df7b3867030aa3a103731 in 899ms, sequenceid=546, compaction requested=true 2023-03-19 09:56:07,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:07,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:07,106 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:07,106 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:07,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:07,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:07,107 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:07,107 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/A is initiating minor compaction (all files) 2023-03-19 09:56:07,107 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:07,107 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/A in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:07,107 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/B is initiating minor compaction (all files) 2023-03-19 09:56:07,107 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d5b3c02ac74044a59ea6d38f85b94f7d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d6b0e2b59fe74d5c87df37185f01f7e9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/35cb3701dbe6464db5dcd44e20be7a18] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=36.8 K 2023-03-19 09:56:07,107 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/B in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:07,108 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03c355234729402a9b5129ddc7e56c7a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e27eaa58bec4426b9559426d3f23a04, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03ded2db284243509c574a30db036d1d] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=36.8 K 2023-03-19 09:56:07,108 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d5b3c02ac74044a59ea6d38f85b94f7d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679219763272 2023-03-19 09:56:07,108 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 03c355234729402a9b5129ddc7e56c7a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679219763272 2023-03-19 09:56:07,108 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d6b0e2b59fe74d5c87df37185f01f7e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679219763970 2023-03-19 09:56:07,108 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4e27eaa58bec4426b9559426d3f23a04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679219763970 2023-03-19 09:56:07,109 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 35cb3701dbe6464db5dcd44e20be7a18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679219766148 2023-03-19 09:56:07,109 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 03ded2db284243509c574a30db036d1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679219766148 2023-03-19 09:56:07,114 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#B#compaction#511 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:07,114 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#A#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:07,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:07,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:56:07,410 DEBUG [Thread-3194] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:54245 2023-03-19 09:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:07,410 DEBUG [Thread-3194] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:07,527 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/5396e18acd39495c9ac493492c38c791 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/5396e18acd39495c9ac493492c38c791 2023-03-19 09:56:07,527 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/0a745342f93a400b9805521b58f0ae2e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0a745342f93a400b9805521b58f0ae2e 2023-03-19 09:56:07,532 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/B of ddd80e01130df7b3867030aa3a103731 into 0a745342f93a400b9805521b58f0ae2e(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:07,532 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/A of ddd80e01130df7b3867030aa3a103731 into 5396e18acd39495c9ac493492c38c791(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:07,532 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:07,532 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:07,532 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/B, priority=13, startTime=1679219767106; duration=0sec 2023-03-19 09:56:07,532 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/A, priority=13, startTime=1679219767106; duration=0sec 2023-03-19 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): ddd80e01130df7b3867030aa3a103731/C is initiating minor compaction (all files) 2023-03-19 09:56:07,533 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ddd80e01130df7b3867030aa3a103731/C in TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:07,534 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9bdedca42848c8a339fe10ba51881f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd6b1f3645044b9aa1d83e4ec80a6a19, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d157656231334e6e9ee7618543a2fc67] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp, totalSize=36.8 K 2023-03-19 09:56:07,534 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5d9bdedca42848c8a339fe10ba51881f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679219763272 2023-03-19 09:56:07,534 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting bd6b1f3645044b9aa1d83e4ec80a6a19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679219763970 2023-03-19 09:56:07,534 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d157656231334e6e9ee7618543a2fc67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679219766148 2023-03-19 09:56:07,539 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): ddd80e01130df7b3867030aa3a103731#C#compaction#513 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:07,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/39f3cda8a44e4bab858cb7e89f218d08 2023-03-19 09:56:07,946 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/301ab36ba5974010bba962ca1bc2aea0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/301ab36ba5974010bba962ca1bc2aea0 2023-03-19 09:56:07,949 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ddd80e01130df7b3867030aa3a103731/C of ddd80e01130df7b3867030aa3a103731 into 301ab36ba5974010bba962ca1bc2aea0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:07,950 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:07,950 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731., storeName=ddd80e01130df7b3867030aa3a103731/C, priority=13, startTime=1679219767106; duration=0sec 2023-03-19 09:56:07,950 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:08,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/6efe008100d249cdb7a22265b86f0361 2023-03-19 09:56:08,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/c0f0b9fa518c4df8925928b3e90f8010 2023-03-19 09:56:08,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/39f3cda8a44e4bab858cb7e89f218d08 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39f3cda8a44e4bab858cb7e89f218d08 2023-03-19 09:56:08,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39f3cda8a44e4bab858cb7e89f218d08, entries=150, sequenceid=562, filesize=11.9 K 2023-03-19 09:56:08,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/6efe008100d249cdb7a22265b86f0361 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6efe008100d249cdb7a22265b86f0361 2023-03-19 09:56:08,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6efe008100d249cdb7a22265b86f0361, entries=150, sequenceid=562, filesize=11.9 K 2023-03-19 09:56:08,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/c0f0b9fa518c4df8925928b3e90f8010 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c0f0b9fa518c4df8925928b3e90f8010 2023-03-19 09:56:08,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c0f0b9fa518c4df8925928b3e90f8010, entries=150, sequenceid=562, filesize=11.9 K 2023-03-19 09:56:08,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for ddd80e01130df7b3867030aa3a103731 in 1276ms, sequenceid=562, compaction requested=false 2023-03-19 09:56:08,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:12,293 DEBUG [Thread-3192] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:54245 2023-03-19 09:56:12,293 DEBUG [Thread-3192] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:12,313 DEBUG [Thread-3188] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:54245 2023-03-19 09:56:12,313 DEBUG [Thread-3188] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:16,812 DEBUG [Thread-3190] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:54245 2023-03-19 09:56:16,813 DEBUG [Thread-3190] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:16,857 DEBUG [Thread-3196] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:54245 2023-03-19 09:56:16,857 DEBUG [Thread-3196] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 245 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-19 09:56:16,857 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1833 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5499 rows 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1825 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5475 rows 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1833 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5499 rows 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1839 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5514 rows 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(402): scanned 1830 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(403): verified 5490 rows 2023-03-19 09:56:16,858 INFO [Listener at localhost/45961] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-19 09:56:16,858 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02df63a4 to 127.0.0.1:54245 2023-03-19 09:56:16,858 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:16,859 INFO [Listener at localhost/45961] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-19 09:56:16,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-19 09:56:16,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-19 09:56:16,864 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219776864"}]},"ts":"1679219776864"} 2023-03-19 09:56:16,865 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-19 09:56:16,866 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-19 09:56:16,867 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ddd80e01130df7b3867030aa3a103731, UNASSIGN}] 2023-03-19 09:56:16,868 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ddd80e01130df7b3867030aa3a103731, UNASSIGN 2023-03-19 09:56:16,869 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ddd80e01130df7b3867030aa3a103731, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:16,869 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219776869"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219776869"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219776869"}]},"ts":"1679219776869"} 2023-03-19 09:56:16,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:56:16,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-19 09:56:17,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:17,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ddd80e01130df7b3867030aa3a103731, disabling compactions & flushes 2023-03-19 09:56:17,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:17,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:17,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. after waiting 0 ms 2023-03-19 09:56:17,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:17,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ddd80e01130df7b3867030aa3a103731 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-19 09:56:17,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=A 2023-03-19 09:56:17,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:17,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=B 2023-03-19 09:56:17,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:17,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ddd80e01130df7b3867030aa3a103731, store=C 2023-03-19 09:56:17,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:17,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-19 09:56:17,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/5c326c591488462da2291173a2feac9e 2023-03-19 09:56:17,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-19 09:56:17,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/5fe15d189eba45489e9d1110d2bfa89c 2023-03-19 09:56:17,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-19 09:56:18,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/1515f8e0352a4fd38634e93339832c7a 2023-03-19 09:56:18,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/A/5c326c591488462da2291173a2feac9e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/5c326c591488462da2291173a2feac9e 2023-03-19 09:56:18,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/5c326c591488462da2291173a2feac9e, entries=100, sequenceid=572, filesize=9.5 K 2023-03-19 09:56:18,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/B/5fe15d189eba45489e9d1110d2bfa89c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/5fe15d189eba45489e9d1110d2bfa89c 2023-03-19 09:56:18,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/5fe15d189eba45489e9d1110d2bfa89c, entries=100, sequenceid=572, filesize=9.5 K 2023-03-19 09:56:18,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/.tmp/C/1515f8e0352a4fd38634e93339832c7a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/1515f8e0352a4fd38634e93339832c7a 2023-03-19 09:56:18,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/1515f8e0352a4fd38634e93339832c7a, entries=100, sequenceid=572, filesize=9.5 K 2023-03-19 09:56:18,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for ddd80e01130df7b3867030aa3a103731 in 1239ms, sequenceid=572, compaction requested=true 2023-03-19 09:56:18,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/27bcd395e3a64d15a1a78a33179d03cd, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/65c07ca9ab6b4aafbc9bfda3cc98c95c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0b256e14282448d3b46bc6a96944e568, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a517eb2b9f0e4b389e7c7c8a82165255, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bd1a801f80a442608e603c434511ba67, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/83825c3d5a8a4e2eae50cfc904a9e10d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/2a522ecaa06a4fd6a8816cfd756d17ec, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/333950f9edb54f1582a38841a023418f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0462a3e58d384d3d822df3c852b929e3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/b67934882a2a44e9a4563cffcd1ea5ba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bc522fd8485341a0a45166da5990063a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f68cb769799341bc8c19c6ebc14f6499, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3c3e2a9df2f64bc5a52c70b3b6a32525, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/16ff2bcb855e4076bbe9a1302c9a2e42, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ebba2eff88f4a7f90dc422dec3561c0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/04f98b64b1bf4d17b132e93ed6e38970, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e1300cb22c68409caa7c6054f118d165, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a6718236c4d7485b9497655ead87be0e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6b6dd8912fe2412ebffa4fbd633a8f36, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/c7ff2cb9a96b4c49a3a906076b487b04, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/706f779929dc41aaa058f6ffe0af1146, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6bef35923f784e7cba62e9695b429b73, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ab590c2d0a94055810d2c527f1ef118, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39667c1c886747e4bbabb4af4c31bc91, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0ee0eacb9b484da8bf8d04d6914486f7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/55b6914fdb7a4e5c990b5ddd32834c6b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/fe3db23c0d5145bcb09614fca0ddfc7a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f45ef1882a7b4d1191b1379b4993d7ba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a532d13bcb9f4cb4aac0f080ba3a8ac1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e266e83259da45f391a237921572a2b6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6aa08c80c43b46f1992283b28246f883, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/ef17193e3ad74e6f95a100d338e5ad76, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/23dd9879bd5c411ba3ed36b99c48ce81, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d5b3c02ac74044a59ea6d38f85b94f7d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d6b0e2b59fe74d5c87df37185f01f7e9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/35cb3701dbe6464db5dcd44e20be7a18] to archive 2023-03-19 09:56:18,286 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:56:18,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/27bcd395e3a64d15a1a78a33179d03cd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/27bcd395e3a64d15a1a78a33179d03cd 2023-03-19 09:56:18,289 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/65c07ca9ab6b4aafbc9bfda3cc98c95c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/65c07ca9ab6b4aafbc9bfda3cc98c95c 2023-03-19 09:56:18,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0b256e14282448d3b46bc6a96944e568 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0b256e14282448d3b46bc6a96944e568 2023-03-19 09:56:18,291 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a517eb2b9f0e4b389e7c7c8a82165255 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a517eb2b9f0e4b389e7c7c8a82165255 2023-03-19 09:56:18,292 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bd1a801f80a442608e603c434511ba67 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bd1a801f80a442608e603c434511ba67 2023-03-19 09:56:18,293 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/83825c3d5a8a4e2eae50cfc904a9e10d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/83825c3d5a8a4e2eae50cfc904a9e10d 2023-03-19 09:56:18,295 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/2a522ecaa06a4fd6a8816cfd756d17ec to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/2a522ecaa06a4fd6a8816cfd756d17ec 2023-03-19 09:56:18,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/333950f9edb54f1582a38841a023418f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/333950f9edb54f1582a38841a023418f 2023-03-19 09:56:18,297 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0462a3e58d384d3d822df3c852b929e3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0462a3e58d384d3d822df3c852b929e3 2023-03-19 09:56:18,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/b67934882a2a44e9a4563cffcd1ea5ba to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/b67934882a2a44e9a4563cffcd1ea5ba 2023-03-19 09:56:18,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bc522fd8485341a0a45166da5990063a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/bc522fd8485341a0a45166da5990063a 2023-03-19 09:56:18,299 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f68cb769799341bc8c19c6ebc14f6499 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f68cb769799341bc8c19c6ebc14f6499 2023-03-19 09:56:18,300 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3c3e2a9df2f64bc5a52c70b3b6a32525 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3c3e2a9df2f64bc5a52c70b3b6a32525 2023-03-19 09:56:18,301 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/16ff2bcb855e4076bbe9a1302c9a2e42 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/16ff2bcb855e4076bbe9a1302c9a2e42 2023-03-19 09:56:18,304 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ebba2eff88f4a7f90dc422dec3561c0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ebba2eff88f4a7f90dc422dec3561c0 2023-03-19 09:56:18,305 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/04f98b64b1bf4d17b132e93ed6e38970 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/04f98b64b1bf4d17b132e93ed6e38970 2023-03-19 09:56:18,305 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e1300cb22c68409caa7c6054f118d165 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e1300cb22c68409caa7c6054f118d165 2023-03-19 09:56:18,306 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a6718236c4d7485b9497655ead87be0e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a6718236c4d7485b9497655ead87be0e 2023-03-19 09:56:18,307 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6b6dd8912fe2412ebffa4fbd633a8f36 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6b6dd8912fe2412ebffa4fbd633a8f36 2023-03-19 09:56:18,308 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/c7ff2cb9a96b4c49a3a906076b487b04 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/c7ff2cb9a96b4c49a3a906076b487b04 2023-03-19 09:56:18,309 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/706f779929dc41aaa058f6ffe0af1146 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/706f779929dc41aaa058f6ffe0af1146 2023-03-19 09:56:18,310 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6bef35923f784e7cba62e9695b429b73 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6bef35923f784e7cba62e9695b429b73 2023-03-19 09:56:18,311 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ab590c2d0a94055810d2c527f1ef118 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/3ab590c2d0a94055810d2c527f1ef118 2023-03-19 09:56:18,312 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39667c1c886747e4bbabb4af4c31bc91 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39667c1c886747e4bbabb4af4c31bc91 2023-03-19 09:56:18,313 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0ee0eacb9b484da8bf8d04d6914486f7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/0ee0eacb9b484da8bf8d04d6914486f7 2023-03-19 09:56:18,314 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/55b6914fdb7a4e5c990b5ddd32834c6b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/55b6914fdb7a4e5c990b5ddd32834c6b 2023-03-19 09:56:18,314 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/fe3db23c0d5145bcb09614fca0ddfc7a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/fe3db23c0d5145bcb09614fca0ddfc7a 2023-03-19 09:56:18,315 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f45ef1882a7b4d1191b1379b4993d7ba to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/f45ef1882a7b4d1191b1379b4993d7ba 2023-03-19 09:56:18,316 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a532d13bcb9f4cb4aac0f080ba3a8ac1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/a532d13bcb9f4cb4aac0f080ba3a8ac1 2023-03-19 09:56:18,317 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e266e83259da45f391a237921572a2b6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/e266e83259da45f391a237921572a2b6 2023-03-19 09:56:18,317 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6aa08c80c43b46f1992283b28246f883 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/6aa08c80c43b46f1992283b28246f883 2023-03-19 09:56:18,318 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/ef17193e3ad74e6f95a100d338e5ad76 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/ef17193e3ad74e6f95a100d338e5ad76 2023-03-19 09:56:18,319 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/23dd9879bd5c411ba3ed36b99c48ce81 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/23dd9879bd5c411ba3ed36b99c48ce81 2023-03-19 09:56:18,319 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d5b3c02ac74044a59ea6d38f85b94f7d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d5b3c02ac74044a59ea6d38f85b94f7d 2023-03-19 09:56:18,320 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d6b0e2b59fe74d5c87df37185f01f7e9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/d6b0e2b59fe74d5c87df37185f01f7e9 2023-03-19 09:56:18,321 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/35cb3701dbe6464db5dcd44e20be7a18 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/35cb3701dbe6464db5dcd44e20be7a18 2023-03-19 09:56:18,324 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/282a5e7b714947528b59c738c01601db, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6d41cd64864244138ea80f37f228a815, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/642b2a3042ed42f7a78634790d36f2f4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/61c29e43303c4825aae200cc720c0c31, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9d11d4a41d7c4fbd957cd74881f4f01f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/7f12c03d9f9544628570a43f2f71ecec, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e1178b6f43064569a58fd44ea5bfb350, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f99b200ef61a42b2a66c91db28023e29, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/424adecf0bc44bd982fae95b80238113, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/c750a5733b764f6e94dd4e2cd7ba6426, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/35ea1663b3d44e12b4c9033ac9fc6019, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/fdd47ff413ac41c5a6c86f19eeb20dba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e851d5d99c14c4e965333c47d303dab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9e6ef826264046829b55458822bc9b0c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2db4a1e5321b40a6b5da876092dead76, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6171c200eb0143a891034fa78fc9e5c3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2df48427866d4f57b4a265b8c742ec41, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/8be2dcfb60934af1b05d568baf7c8761, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d1126312d8ad45eda4b05b75068cf4ee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f3529f291be44400b04e84212489461c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4900e7f461e842f19c02441b60310d19, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/63432edb60a54a1eb5ee97d1b3c2349b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e364c395a1104bbfa58a875aa945e416, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/de8a29166e08433e99addead96ec92ad, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0d4475b3f0644f699ce390c6a9db2daf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/cffe891bc3f44b7e9352258ecc89d352, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/ac9506d851194b37b197653efb831c9c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/83b16148cd804b4f95345132d423d582, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2cc44c8b950146f386b7c4c39cb17b0d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/965a6c6fb80a476385bb6c31479279ab, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d01d2317bedf4c31b291050defd024f4, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03c355234729402a9b5129ddc7e56c7a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/47684f7a549a4cc0ba313a49e795b37f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e27eaa58bec4426b9559426d3f23a04, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03ded2db284243509c574a30db036d1d] to archive 2023-03-19 09:56:18,324 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:56:18,326 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/282a5e7b714947528b59c738c01601db to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/282a5e7b714947528b59c738c01601db 2023-03-19 09:56:18,326 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6d41cd64864244138ea80f37f228a815 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6d41cd64864244138ea80f37f228a815 2023-03-19 09:56:18,327 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/642b2a3042ed42f7a78634790d36f2f4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/642b2a3042ed42f7a78634790d36f2f4 2023-03-19 09:56:18,328 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/61c29e43303c4825aae200cc720c0c31 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/61c29e43303c4825aae200cc720c0c31 2023-03-19 09:56:18,329 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9d11d4a41d7c4fbd957cd74881f4f01f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9d11d4a41d7c4fbd957cd74881f4f01f 2023-03-19 09:56:18,330 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/7f12c03d9f9544628570a43f2f71ecec to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/7f12c03d9f9544628570a43f2f71ecec 2023-03-19 09:56:18,331 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e1178b6f43064569a58fd44ea5bfb350 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e1178b6f43064569a58fd44ea5bfb350 2023-03-19 09:56:18,332 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f99b200ef61a42b2a66c91db28023e29 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f99b200ef61a42b2a66c91db28023e29 2023-03-19 09:56:18,334 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/424adecf0bc44bd982fae95b80238113 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/424adecf0bc44bd982fae95b80238113 2023-03-19 09:56:18,335 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/c750a5733b764f6e94dd4e2cd7ba6426 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/c750a5733b764f6e94dd4e2cd7ba6426 2023-03-19 09:56:18,336 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/35ea1663b3d44e12b4c9033ac9fc6019 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/35ea1663b3d44e12b4c9033ac9fc6019 2023-03-19 09:56:18,337 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/fdd47ff413ac41c5a6c86f19eeb20dba to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/fdd47ff413ac41c5a6c86f19eeb20dba 2023-03-19 09:56:18,338 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e851d5d99c14c4e965333c47d303dab to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e851d5d99c14c4e965333c47d303dab 2023-03-19 09:56:18,339 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9e6ef826264046829b55458822bc9b0c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/9e6ef826264046829b55458822bc9b0c 2023-03-19 09:56:18,340 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2db4a1e5321b40a6b5da876092dead76 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2db4a1e5321b40a6b5da876092dead76 2023-03-19 09:56:18,341 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6171c200eb0143a891034fa78fc9e5c3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6171c200eb0143a891034fa78fc9e5c3 2023-03-19 09:56:18,342 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2df48427866d4f57b4a265b8c742ec41 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2df48427866d4f57b4a265b8c742ec41 2023-03-19 09:56:18,343 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/8be2dcfb60934af1b05d568baf7c8761 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/8be2dcfb60934af1b05d568baf7c8761 2023-03-19 09:56:18,344 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d1126312d8ad45eda4b05b75068cf4ee to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d1126312d8ad45eda4b05b75068cf4ee 2023-03-19 09:56:18,345 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f3529f291be44400b04e84212489461c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/f3529f291be44400b04e84212489461c 2023-03-19 09:56:18,346 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4900e7f461e842f19c02441b60310d19 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4900e7f461e842f19c02441b60310d19 2023-03-19 09:56:18,347 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/63432edb60a54a1eb5ee97d1b3c2349b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/63432edb60a54a1eb5ee97d1b3c2349b 2023-03-19 09:56:18,348 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e364c395a1104bbfa58a875aa945e416 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/e364c395a1104bbfa58a875aa945e416 2023-03-19 09:56:18,349 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/de8a29166e08433e99addead96ec92ad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/de8a29166e08433e99addead96ec92ad 2023-03-19 09:56:18,350 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0d4475b3f0644f699ce390c6a9db2daf to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0d4475b3f0644f699ce390c6a9db2daf 2023-03-19 09:56:18,351 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/cffe891bc3f44b7e9352258ecc89d352 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/cffe891bc3f44b7e9352258ecc89d352 2023-03-19 09:56:18,357 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/ac9506d851194b37b197653efb831c9c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/ac9506d851194b37b197653efb831c9c 2023-03-19 09:56:18,358 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0ccf4a0929ea4bdf8c07ec27e8dd5a2d 2023-03-19 09:56:18,359 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/83b16148cd804b4f95345132d423d582 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/83b16148cd804b4f95345132d423d582 2023-03-19 09:56:18,360 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2cc44c8b950146f386b7c4c39cb17b0d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/2cc44c8b950146f386b7c4c39cb17b0d 2023-03-19 09:56:18,361 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/965a6c6fb80a476385bb6c31479279ab to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/965a6c6fb80a476385bb6c31479279ab 2023-03-19 09:56:18,362 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d01d2317bedf4c31b291050defd024f4 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/d01d2317bedf4c31b291050defd024f4 2023-03-19 09:56:18,363 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03c355234729402a9b5129ddc7e56c7a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03c355234729402a9b5129ddc7e56c7a 2023-03-19 09:56:18,364 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/47684f7a549a4cc0ba313a49e795b37f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/47684f7a549a4cc0ba313a49e795b37f 2023-03-19 09:56:18,366 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e27eaa58bec4426b9559426d3f23a04 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/4e27eaa58bec4426b9559426d3f23a04 2023-03-19 09:56:18,367 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03ded2db284243509c574a30db036d1d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/03ded2db284243509c574a30db036d1d 2023-03-19 09:56:18,370 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c3f28305197f47b88ef375a5c59707da, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/664253bc14ce4a09a3dacb08524eec06, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5cb128a8f6804a4b90a1e53bb95a3c29, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a7341af58e8d465ebc6e332a2a85f642, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/987be4fe581c46dfa6cad18801800b5a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2e7630b1417d42fabb7ee4cca2be77f3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/747727899350441b84aca9f64d12af5a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2b09132417294074a630abd3707aed60, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/b74e3da5c11f4aa8b7904de967f97c73, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4135c80123694cff9c2b1ebb60009c9f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/94848d579c5741898d05fb5502d9b248, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/935eade305d94ac081f537ca89adb054, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d7f8b1f873974c4fb1e309c38775375c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/f938e509ba9640028b40ca4b850592d9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/773d22000f894dd688b94505b413119a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9beae549704ba7b91eb1168d6ddd5f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8f610fdd4ea344109b948cbaab72b2d6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/34cd6be951dd4571a569d3f6b6757e37, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd4e12090dea4cacbb9dde033d9abf74, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/6e605770e2ca453ba8d04ddf59b5df8b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a26ee9ca865c4f498a5efb983d08211d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/77ab796f8812400fbe43d1e283d81688, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d2d4ade747484c8cb0bcbc7996894087, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/47f4a7298ce74992b504b7019d88be18, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5247d93b303a40d59f8562bead17e1fe, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5269437854b64a3cbeb9ef0f05dd0dfa, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4e726915bac74526800ab67fb228c7a8, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8948cdfdef3648bdaac3b9b6f2d00ba1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5bf575fb303d4ab080e95dc6827f1c6a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/cd1c066948b74761a441e1e4f7e7e274, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d94c866f4e3f462993bc51044c6b6c16, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/dfe3d9ccd92c4968a07d38316abb787d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9bdedca42848c8a339fe10ba51881f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/88eb6b87763c44ee9bb1433a6530615d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd6b1f3645044b9aa1d83e4ec80a6a19, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d157656231334e6e9ee7618543a2fc67] to archive 2023-03-19 09:56:18,370 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:56:18,372 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c3f28305197f47b88ef375a5c59707da to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c3f28305197f47b88ef375a5c59707da 2023-03-19 09:56:18,373 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/664253bc14ce4a09a3dacb08524eec06 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/664253bc14ce4a09a3dacb08524eec06 2023-03-19 09:56:18,374 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5cb128a8f6804a4b90a1e53bb95a3c29 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5cb128a8f6804a4b90a1e53bb95a3c29 2023-03-19 09:56:18,375 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a7341af58e8d465ebc6e332a2a85f642 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a7341af58e8d465ebc6e332a2a85f642 2023-03-19 09:56:18,376 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/987be4fe581c46dfa6cad18801800b5a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/987be4fe581c46dfa6cad18801800b5a 2023-03-19 09:56:18,377 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2e7630b1417d42fabb7ee4cca2be77f3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2e7630b1417d42fabb7ee4cca2be77f3 2023-03-19 09:56:18,378 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/747727899350441b84aca9f64d12af5a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/747727899350441b84aca9f64d12af5a 2023-03-19 09:56:18,379 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2b09132417294074a630abd3707aed60 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/2b09132417294074a630abd3707aed60 2023-03-19 09:56:18,380 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/b74e3da5c11f4aa8b7904de967f97c73 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/b74e3da5c11f4aa8b7904de967f97c73 2023-03-19 09:56:18,381 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4135c80123694cff9c2b1ebb60009c9f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4135c80123694cff9c2b1ebb60009c9f 2023-03-19 09:56:18,382 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/94848d579c5741898d05fb5502d9b248 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/94848d579c5741898d05fb5502d9b248 2023-03-19 09:56:18,383 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/935eade305d94ac081f537ca89adb054 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/935eade305d94ac081f537ca89adb054 2023-03-19 09:56:18,384 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d7f8b1f873974c4fb1e309c38775375c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d7f8b1f873974c4fb1e309c38775375c 2023-03-19 09:56:18,385 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/f938e509ba9640028b40ca4b850592d9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/f938e509ba9640028b40ca4b850592d9 2023-03-19 09:56:18,387 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/773d22000f894dd688b94505b413119a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/773d22000f894dd688b94505b413119a 2023-03-19 09:56:18,388 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9beae549704ba7b91eb1168d6ddd5f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9beae549704ba7b91eb1168d6ddd5f 2023-03-19 09:56:18,389 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8f610fdd4ea344109b948cbaab72b2d6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8f610fdd4ea344109b948cbaab72b2d6 2023-03-19 09:56:18,390 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/34cd6be951dd4571a569d3f6b6757e37 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/34cd6be951dd4571a569d3f6b6757e37 2023-03-19 09:56:18,391 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd4e12090dea4cacbb9dde033d9abf74 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd4e12090dea4cacbb9dde033d9abf74 2023-03-19 09:56:18,392 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/6e605770e2ca453ba8d04ddf59b5df8b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/6e605770e2ca453ba8d04ddf59b5df8b 2023-03-19 09:56:18,393 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a26ee9ca865c4f498a5efb983d08211d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/a26ee9ca865c4f498a5efb983d08211d 2023-03-19 09:56:18,394 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/77ab796f8812400fbe43d1e283d81688 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/77ab796f8812400fbe43d1e283d81688 2023-03-19 09:56:18,394 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d2d4ade747484c8cb0bcbc7996894087 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d2d4ade747484c8cb0bcbc7996894087 2023-03-19 09:56:18,395 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/47f4a7298ce74992b504b7019d88be18 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/47f4a7298ce74992b504b7019d88be18 2023-03-19 09:56:18,396 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5247d93b303a40d59f8562bead17e1fe to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5247d93b303a40d59f8562bead17e1fe 2023-03-19 09:56:18,397 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5269437854b64a3cbeb9ef0f05dd0dfa to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5269437854b64a3cbeb9ef0f05dd0dfa 2023-03-19 09:56:18,397 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4e726915bac74526800ab67fb228c7a8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/4e726915bac74526800ab67fb228c7a8 2023-03-19 09:56:18,398 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8948cdfdef3648bdaac3b9b6f2d00ba1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/8948cdfdef3648bdaac3b9b6f2d00ba1 2023-03-19 09:56:18,399 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5bf575fb303d4ab080e95dc6827f1c6a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5bf575fb303d4ab080e95dc6827f1c6a 2023-03-19 09:56:18,399 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/cd1c066948b74761a441e1e4f7e7e274 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/cd1c066948b74761a441e1e4f7e7e274 2023-03-19 09:56:18,400 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d94c866f4e3f462993bc51044c6b6c16 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d94c866f4e3f462993bc51044c6b6c16 2023-03-19 09:56:18,401 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/dfe3d9ccd92c4968a07d38316abb787d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/dfe3d9ccd92c4968a07d38316abb787d 2023-03-19 09:56:18,401 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9bdedca42848c8a339fe10ba51881f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/5d9bdedca42848c8a339fe10ba51881f 2023-03-19 09:56:18,402 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/88eb6b87763c44ee9bb1433a6530615d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/88eb6b87763c44ee9bb1433a6530615d 2023-03-19 09:56:18,403 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd6b1f3645044b9aa1d83e4ec80a6a19 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/bd6b1f3645044b9aa1d83e4ec80a6a19 2023-03-19 09:56:18,403 DEBUG [StoreCloser-TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d157656231334e6e9ee7618543a2fc67 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/d157656231334e6e9ee7618543a2fc67 2023-03-19 09:56:18,406 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/recovered.edits/575.seqid, newMaxSeqId=575, maxSeqId=1 2023-03-19 09:56:18,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731. 2023-03-19 09:56:18,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ddd80e01130df7b3867030aa3a103731: 2023-03-19 09:56:18,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:18,410 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ddd80e01130df7b3867030aa3a103731, regionState=CLOSED 2023-03-19 09:56:18,410 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219778410"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219778410"}]},"ts":"1679219778410"} 2023-03-19 09:56:18,412 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-19 09:56:18,412 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure ddd80e01130df7b3867030aa3a103731, server=jenkins-hbase4.apache.org,34669,1679219618625 in 1.5420 sec 2023-03-19 09:56:18,413 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-19 09:56:18,413 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ddd80e01130df7b3867030aa3a103731, UNASSIGN in 1.5450 sec 2023-03-19 09:56:18,413 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219778413"}]},"ts":"1679219778413"} 2023-03-19 09:56:18,414 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-19 09:56:18,415 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-19 09:56:18,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5560 sec 2023-03-19 09:56:18,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-19 09:56:18,968 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-19 09:56:18,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-19 09:56:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:18,971 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:18,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-19 09:56:18,971 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:18,976 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:18,978 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/recovered.edits] 2023-03-19 09:56:18,980 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39f3cda8a44e4bab858cb7e89f218d08 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/39f3cda8a44e4bab858cb7e89f218d08 2023-03-19 09:56:18,981 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/5396e18acd39495c9ac493492c38c791 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/5396e18acd39495c9ac493492c38c791 2023-03-19 09:56:18,982 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/5c326c591488462da2291173a2feac9e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/A/5c326c591488462da2291173a2feac9e 2023-03-19 09:56:18,983 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0a745342f93a400b9805521b58f0ae2e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/0a745342f93a400b9805521b58f0ae2e 2023-03-19 09:56:18,984 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/5fe15d189eba45489e9d1110d2bfa89c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/5fe15d189eba45489e9d1110d2bfa89c 2023-03-19 09:56:18,985 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6efe008100d249cdb7a22265b86f0361 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/B/6efe008100d249cdb7a22265b86f0361 2023-03-19 09:56:18,987 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/1515f8e0352a4fd38634e93339832c7a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/1515f8e0352a4fd38634e93339832c7a 2023-03-19 09:56:18,988 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/301ab36ba5974010bba962ca1bc2aea0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/301ab36ba5974010bba962ca1bc2aea0 2023-03-19 09:56:18,989 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c0f0b9fa518c4df8925928b3e90f8010 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/C/c0f0b9fa518c4df8925928b3e90f8010 2023-03-19 09:56:18,991 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/recovered.edits/575.seqid to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731/recovered.edits/575.seqid 2023-03-19 09:56:18,991 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/ddd80e01130df7b3867030aa3a103731 2023-03-19 09:56:18,991 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:56:18,993 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:18,994 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-19 09:56:18,995 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-19 09:56:18,996 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:18,996 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-19 09:56:18,996 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679219778996"}]},"ts":"9223372036854775807"} 2023-03-19 09:56:18,998 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-19 09:56:18,998 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ddd80e01130df7b3867030aa3a103731, NAME => 'TestAcidGuarantees,,1679219745180.ddd80e01130df7b3867030aa3a103731.', STARTKEY => '', ENDKEY => ''}] 2023-03-19 09:56:18,998 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-19 09:56:18,998 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679219778998"}]},"ts":"9223372036854775807"} 2023-03-19 09:56:18,999 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-19 09:56:19,001 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:19,001 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-03-19 09:56:19,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-19 09:56:19,072 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-19 09:56:19,083 INFO [Listener at localhost/45961] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=524 (was 507) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=342 (was 314) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1739 (was 1386) - AvailableMemoryMB LEAK? - 2023-03-19 09:56:19,095 INFO [Listener at localhost/45961] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=342, ProcessCount=169, AvailableMemoryMB=1738 2023-03-19 09:56:19,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:56:19,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:56:19,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:19,099 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-19 09:56:19,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-19 09:56:19,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-19 09:56:19,100 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-19 09:56:19,101 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:19,102 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 empty. 2023-03-19 09:56:19,102 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:19,102 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:56:19,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-19 09:56:19,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-19 09:56:19,508 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:56:19,510 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a391bedbca383dedc4bc8fc94dd92901, NAME => 'TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp 2023-03-19 09:56:19,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-19 09:56:19,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:56:19,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a391bedbca383dedc4bc8fc94dd92901, disabling compactions & flushes 2023-03-19 09:56:19,917 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:19,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:19,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. after waiting 0 ms 2023-03-19 09:56:19,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:19,917 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:19,917 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:19,919 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-19 09:56:19,920 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219779920"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219779920"}]},"ts":"1679219779920"} 2023-03-19 09:56:19,921 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-19 09:56:19,921 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-19 09:56:19,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219779921"}]},"ts":"1679219779921"} 2023-03-19 09:56:19,922 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-19 09:56:19,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, ASSIGN}] 2023-03-19 09:56:19,931 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, ASSIGN 2023-03-19 09:56:19,931 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=false 2023-03-19 09:56:20,082 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:20,082 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219780082"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219780082"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219780082"}]},"ts":"1679219780082"} 2023-03-19 09:56:20,084 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:56:20,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-19 09:56:20,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:20,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a391bedbca383dedc4bc8fc94dd92901, NAME => 'TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:56:20,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:56:20,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,239 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,240 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A 2023-03-19 09:56:20,240 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A 2023-03-19 09:56:20,240 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:56:20,240 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a391bedbca383dedc4bc8fc94dd92901 columnFamilyName A 2023-03-19 09:56:20,241 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(310): Store=a391bedbca383dedc4bc8fc94dd92901/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:56:20,241 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,242 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B 2023-03-19 09:56:20,242 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B 2023-03-19 09:56:20,242 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:56:20,242 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a391bedbca383dedc4bc8fc94dd92901 columnFamilyName B 2023-03-19 09:56:20,242 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(310): Store=a391bedbca383dedc4bc8fc94dd92901/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:56:20,242 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,243 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C 2023-03-19 09:56:20,243 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C 2023-03-19 09:56:20,243 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:56:20,243 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a391bedbca383dedc4bc8fc94dd92901 columnFamilyName C 2023-03-19 09:56:20,244 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(310): Store=a391bedbca383dedc4bc8fc94dd92901/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:56:20,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:20,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:56:20,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:20,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-19 09:56:20,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a391bedbca383dedc4bc8fc94dd92901; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70506938, jitterRate=0.05063524842262268}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:56:20,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:20,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., pid=56, masterSystemTime=1679219780235 2023-03-19 09:56:20,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:20,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:20,250 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:20,250 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219780250"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219780250"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219780250"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219780250"}]},"ts":"1679219780250"} 2023-03-19 09:56:20,252 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-19 09:56:20,252 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 in 168 msec 2023-03-19 09:56:20,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-19 09:56:20,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, ASSIGN in 323 msec 2023-03-19 09:56:20,254 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-19 09:56:20,254 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219780254"}]},"ts":"1679219780254"} 2023-03-19 09:56:20,254 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-19 09:56:20,256 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-19 09:56:20,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-03-19 09:56:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-19 09:56:21,205 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-19 09:56:21,205 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x68067de1 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:21,211 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@470742aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:21,212 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:21,214 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36708, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:21,215 DEBUG [Listener at localhost/45961] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-19 09:56:21,216 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-19 09:56:21,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-19 09:56:21,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-19 09:56:21,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:21,627 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-19 09:56:21,635 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-19 09:56:21,635 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-19 09:56:21,638 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-19 09:56:21,641 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, REOPEN/MOVE}] 2023-03-19 09:56:21,641 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, REOPEN/MOVE 2023-03-19 09:56:21,642 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:21,642 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219781642"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219781642"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219781642"}]},"ts":"1679219781642"} 2023-03-19 09:56:21,643 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:56:21,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:21,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a391bedbca383dedc4bc8fc94dd92901, disabling compactions & flushes 2023-03-19 09:56:21,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:21,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:21,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. after waiting 0 ms 2023-03-19 09:56:21,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:21,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-19 09:56:21,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:21,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:21,800 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a391bedbca383dedc4bc8fc94dd92901 to self. 2023-03-19 09:56:21,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:21,802 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=CLOSED 2023-03-19 09:56:21,802 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219781802"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219781802"}]},"ts":"1679219781802"} 2023-03-19 09:56:21,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-19 09:56:21,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 in 160 msec 2023-03-19 09:56:21,805 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34669,1679219618625; forceNewPlan=false, retain=true 2023-03-19 09:56:21,955 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:21,955 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219781955"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219781955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219781955"}]},"ts":"1679219781955"} 2023-03-19 09:56:21,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:56:22,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a391bedbca383dedc4bc8fc94dd92901, NAME => 'TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.', STARTKEY => '', ENDKEY => ''} 2023-03-19 09:56:22,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-19 09:56:22,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,116 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,117 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A 2023-03-19 09:56:22,117 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A 2023-03-19 09:56:22,117 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:56:22,117 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a391bedbca383dedc4bc8fc94dd92901 columnFamilyName A 2023-03-19 09:56:22,119 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(310): Store=a391bedbca383dedc4bc8fc94dd92901/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:56:22,119 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,120 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B 2023-03-19 09:56:22,120 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B 2023-03-19 09:56:22,120 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:56:22,121 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a391bedbca383dedc4bc8fc94dd92901 columnFamilyName B 2023-03-19 09:56:22,121 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(310): Store=a391bedbca383dedc4bc8fc94dd92901/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:56:22,121 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,122 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C 2023-03-19 09:56:22,122 DEBUG [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C 2023-03-19 09:56:22,123 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-19 09:56:22,123 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a391bedbca383dedc4bc8fc94dd92901 columnFamilyName C 2023-03-19 09:56:22,123 INFO [StoreOpener-a391bedbca383dedc4bc8fc94dd92901-1] regionserver.HStore(310): Store=a391bedbca383dedc4bc8fc94dd92901/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-19 09:56:22,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-19 09:56:22,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a391bedbca383dedc4bc8fc94dd92901; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72817183, jitterRate=0.08506058156490326}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-19 09:56:22,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:22,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., pid=61, masterSystemTime=1679219782108 2023-03-19 09:56:22,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,131 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,132 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219782131"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679219782131"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679219782131"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679219782131"}]},"ts":"1679219782131"} 2023-03-19 09:56:22,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-19 09:56:22,134 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 in 176 msec 2023-03-19 09:56:22,135 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-19 09:56:22,135 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, REOPEN/MOVE in 493 msec 2023-03-19 09:56:22,137 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-19 09:56:22,137 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 497 msec 2023-03-19 09:56:22,138 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 919 msec 2023-03-19 09:56:22,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-19 09:56:22,140 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,148 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,149 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,157 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,157 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,165 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,165 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,175 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,175 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,180 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,180 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,189 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,190 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,196 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,197 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,208 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,209 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,219 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,220 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:54245 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-19 09:56:22,228 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-19 09:56:22,233 DEBUG [hconnection-0x3e47483b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,244 DEBUG [hconnection-0x2abeeea9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:22,252 DEBUG [hconnection-0x17af9f94-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,252 DEBUG [hconnection-0x60b25dab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,252 DEBUG [hconnection-0x29ef5a04-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,252 DEBUG [hconnection-0x49baa796-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,253 DEBUG [hconnection-0x3e0f6201-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,254 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36714, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,254 DEBUG [hconnection-0x13db7840-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,256 DEBUG [hconnection-0x61456402-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,259 DEBUG [hconnection-0x23c066a9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-19 09:56:22,260 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36710, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,261 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36720, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,262 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36730, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,262 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36746, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,263 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36740, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,263 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36762, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,263 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36760, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:22,272 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36752, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:22,273 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36776, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-19 09:56:22,273 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:22,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:22,273 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:22,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,275 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,275 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,276 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,276 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:22,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:22,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:22,289 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:22,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-19 09:56:22,293 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,293 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:22,293 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:22,294 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,296 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378b05aa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:22,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:22,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,300 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:22,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:22,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:22,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,300 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:22,302 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:22,305 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:22,305 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:22,305 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:22,305 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:22,305 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:22,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,309 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219842312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219842312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219842312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219842313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219842314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,317 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,317 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:22,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:22,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,318 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:22,318 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:22,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219842415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219842415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219842416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219842416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219842417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,418 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:22,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:22,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:22,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:22,425 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:22,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:22,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:22,425 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:22,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:22,430 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,430 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:22,430 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:22,430 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:22,431 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:22,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:22,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2bc191[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:22,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:22,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:22,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:22,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:22,440 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:22,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,440 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:22,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:22,445 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:22,445 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:22,445 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:22,445 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:22,445 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:22,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,454 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,455 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,459 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,460 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:22,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:22,460 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,460 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:22,461 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:22,561 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:22,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:22,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:22,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:22,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:22,573 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:22,573 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,573 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:22,573 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:22,574 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,574 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,576 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,577 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,577 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:22,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:22,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:22,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:22,578 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,578 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:22,578 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:22,578 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:22,578 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:22,580 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,580 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dbf5c3[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:22,581 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:22,581 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,582 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:22,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:22,583 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:22,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:22,587 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:22,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:22,590 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:22,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:22,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:22,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:22,590 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:22,592 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,592 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,592 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,594 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:22,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:22,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,596 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,597 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:22,597 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:22,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,597 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,597 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:22,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219842618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219842618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219842618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219842619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219842619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,697 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:22,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:22,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:22,716 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:22,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:22,716 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:22,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,717 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,718 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,718 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:22,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:22,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:22,723 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:22,723 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:22,723 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:22,723 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:22,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:22,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26022d79[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:22,727 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:22,727 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,729 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:22,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:22,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:22,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,730 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:22,730 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319a5a9dad56f0e41e79c23962efdf03b87 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a5a9dad56f0e41e79c23962efdf03b87 2023-03-19 09:56:22,731 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:22,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,731 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:22,735 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:22,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:22,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:22,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:22,735 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:22,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,737 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,736 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,740 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,743 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:22,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:22,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:22,744 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:22,744 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:22,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,844 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:22,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:22,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:22,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:22,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:22,848 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:22,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:22,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:22,848 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,849 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,850 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,850 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:22,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:22,863 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:22,863 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:22,864 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:22,876 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:22,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cbca46[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:22,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:22,878 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,880 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:22,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:22,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:22,881 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:22,883 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:22,885 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:22,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:22,885 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:22,886 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:22,886 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:22,887 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,887 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,887 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:22,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:22,889 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:22,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:22,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:22,892 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:22,891 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:22,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:22,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:22,892 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:22,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:22,892 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:22,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:22,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219842922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219842922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219842922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219842923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:22,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:22,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219842923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,000 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:23,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:23,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:23,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:23,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:23,009 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:23,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:23,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:23,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,011 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:23,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,011 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:23,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:23,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:23,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,013 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:23,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:23,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:23,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:23,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:23,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:23,016 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:23,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:23,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:23,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:23,016 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:23,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,018 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:23,020 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:23,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d68bda7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:23,020 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,022 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:23,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:23,024 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:23,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:23,027 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:23,027 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:23,027 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:23,027 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:23,027 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:23,029 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,029 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,029 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:23,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,029 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,038 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:23,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:23,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:23,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,038 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:23,038 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:23,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,038 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,044 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:23,044 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:23,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:23,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:23,138 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:23,139 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=19, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/49f819207a954b3e819c5a2aecae0cc7 2023-03-19 09:56:23,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:23,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:23,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:23,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:23,148 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:23,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:23,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:23,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,148 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,162 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:23,162 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:23,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:23,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:23,185 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:23,185 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:23,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/14d4323f727c4be5989b896ec5b282df 2023-03-19 09:56:23,185 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:23,185 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:23,186 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:23,186 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:23,187 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:23,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42122c24[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:23,190 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:23,190 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,191 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:23,191 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,191 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:23,194 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:23,200 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:23,200 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:23,200 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:23,200 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:23,200 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:23,203 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,203 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,203 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:23,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,204 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,206 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:23,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:23,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:23,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:23,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:23,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:23,210 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,211 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,211 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,211 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:23,211 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:23,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:23,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/fd16037091d446ff9a15f4a82219210c 2023-03-19 09:56:23,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/49f819207a954b3e819c5a2aecae0cc7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/49f819207a954b3e819c5a2aecae0cc7 2023-03-19 09:56:23,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/49f819207a954b3e819c5a2aecae0cc7, entries=150, sequenceid=19, filesize=25.1 K 2023-03-19 09:56:23,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/14d4323f727c4be5989b896ec5b282df as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/14d4323f727c4be5989b896ec5b282df 2023-03-19 09:56:23,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/14d4323f727c4be5989b896ec5b282df, entries=150, sequenceid=19, filesize=11.6 K 2023-03-19 09:56:23,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/fd16037091d446ff9a15f4a82219210c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fd16037091d446ff9a15f4a82219210c 2023-03-19 09:56:23,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fd16037091d446ff9a15f4a82219210c, entries=150, sequenceid=19, filesize=11.6 K 2023-03-19 09:56:23,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a391bedbca383dedc4bc8fc94dd92901 in 942ms, sequenceid=19, compaction requested=false 2023-03-19 09:56:23,235 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-19 09:56:23,235 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:23,311 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:23,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:23,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:23,314 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:23,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:23,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:23,314 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,316 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:23,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:23,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:23,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:23,319 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:23,319 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:23,319 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:23,319 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:23,320 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-19 09:56:23,320 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:23,320 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:23,320 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:23,320 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:23,320 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:23,320 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:23,331 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303195c4686fb6d4b4cd683cbc6e1c3afd590 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195c4686fb6d4b4cd683cbc6e1c3afd590 2023-03-19 09:56:23,337 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/93feb0764c4a4256a2587e18bd7865c0 2023-03-19 09:56:23,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:23,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:23,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219843437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219843437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219843440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219843440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219843441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219843541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219843542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219843545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219843545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219843545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,745 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/438ad338bc1346aabfb2fe28af0cd870 2023-03-19 09:56:23,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219843745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219843745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219843748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219843748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:23,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219843749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,754 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/6b70e918ac7245c19f86a5a69a4d70ad 2023-03-19 09:56:23,757 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/93feb0764c4a4256a2587e18bd7865c0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/93feb0764c4a4256a2587e18bd7865c0 2023-03-19 09:56:23,760 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/93feb0764c4a4256a2587e18bd7865c0, entries=150, sequenceid=36, filesize=25.1 K 2023-03-19 09:56:23,761 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/438ad338bc1346aabfb2fe28af0cd870 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/438ad338bc1346aabfb2fe28af0cd870 2023-03-19 09:56:23,775 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/438ad338bc1346aabfb2fe28af0cd870, entries=150, sequenceid=36, filesize=11.6 K 2023-03-19 09:56:23,776 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/6b70e918ac7245c19f86a5a69a4d70ad as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/6b70e918ac7245c19f86a5a69a4d70ad 2023-03-19 09:56:23,784 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/6b70e918ac7245c19f86a5a69a4d70ad, entries=150, sequenceid=36, filesize=11.6 K 2023-03-19 09:56:23,785 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for a391bedbca383dedc4bc8fc94dd92901 in 465ms, sequenceid=36, compaction requested=false 2023-03-19 09:56:23,785 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:23,785 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:23,787 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75cca8fa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:23,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:23,790 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:23,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:23,791 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:23,791 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,791 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:23,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:23,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:23,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:23,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:23,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:23,797 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:23,797 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:23,797 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:23,797 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:23,797 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:23,799 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,799 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:23,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,799 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:23,800 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,802 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:23,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:23,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:23,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:23,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:23,807 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:23,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:23,807 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:23,808 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:23,808 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:23,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:23,808 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,908 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:23,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:23,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:23,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:23,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:23,917 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:23,917 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,917 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:23,917 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:23,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,920 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:23,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:23,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:23,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:23,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:23,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:23,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:23,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:23,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:23,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:23,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:23,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:23,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:23,927 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:23,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:23,927 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:23,928 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:56:23,928 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:23,928 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:23,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:23,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:23,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:23,929 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:23,942 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-19 09:56:24,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:24,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:24,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219844058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219844059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219844059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219844060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219844060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219844161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219844163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219844164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219844164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219844164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,346 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031936dd5bd0d7da489ab523a6416ceee0ea to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031936dd5bd0d7da489ab523a6416ceee0ea 2023-03-19 09:56:24,364 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/5940edeb7f764922bd26b689e3de117c 2023-03-19 09:56:24,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219844364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219844366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219844367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219844368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219844368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219844668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219844670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219844672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219844673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:24,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219844674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:24,787 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/6c1f3d1782b04947944803a78db7bfeb 2023-03-19 09:56:25,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:25,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219845170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:25,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219845175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:25,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219845176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219845179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219845180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,196 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/becc227127f24f6b825701f8bbca7053 2023-03-19 09:56:25,200 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/5940edeb7f764922bd26b689e3de117c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5940edeb7f764922bd26b689e3de117c 2023-03-19 09:56:25,203 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5940edeb7f764922bd26b689e3de117c, entries=150, sequenceid=52, filesize=25.1 K 2023-03-19 09:56:25,204 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/6c1f3d1782b04947944803a78db7bfeb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6c1f3d1782b04947944803a78db7bfeb 2023-03-19 09:56:25,208 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6c1f3d1782b04947944803a78db7bfeb, entries=150, sequenceid=52, filesize=11.6 K 2023-03-19 09:56:25,209 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/becc227127f24f6b825701f8bbca7053 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/becc227127f24f6b825701f8bbca7053 2023-03-19 09:56:25,212 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/becc227127f24f6b825701f8bbca7053, entries=150, sequenceid=52, filesize=11.6 K 2023-03-19 09:56:25,213 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for a391bedbca383dedc4bc8fc94dd92901 in 1285ms, sequenceid=52, compaction requested=true 2023-03-19 09:56:25,213 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:25,213 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,215 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:25,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c2d5ff[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:25,218 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:25,218 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,221 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,221 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:25,223 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,223 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:25,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:25,227 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:25,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:25,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:25,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:25,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:25,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,230 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,231 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,232 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:25,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,236 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:25,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:25,237 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:25,237 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:25,236 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,237 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,337 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:25,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:25,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:25,340 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:25,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:25,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:25,341 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,341 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,342 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:25,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:25,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:25,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,344 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:25,344 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-19 09:56:25,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:25,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:25,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:25,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:25,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:25,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:25,357 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303193b15110eeaca47ff92cb6374c47fd4da to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193b15110eeaca47ff92cb6374c47fd4da 2023-03-19 09:56:25,363 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/3f0906d6af6c4bac8d9ee4330237c2d3 2023-03-19 09:56:25,382 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/60922ab72b6c4917b5063009c15c225c 2023-03-19 09:56:25,395 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/c4143803a85d47e7b16a94b8e941bcbf 2023-03-19 09:56:25,398 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/3f0906d6af6c4bac8d9ee4330237c2d3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/3f0906d6af6c4bac8d9ee4330237c2d3 2023-03-19 09:56:25,406 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/3f0906d6af6c4bac8d9ee4330237c2d3, entries=150, sequenceid=68, filesize=25.1 K 2023-03-19 09:56:25,407 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/60922ab72b6c4917b5063009c15c225c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/60922ab72b6c4917b5063009c15c225c 2023-03-19 09:56:25,421 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/60922ab72b6c4917b5063009c15c225c, entries=150, sequenceid=68, filesize=11.6 K 2023-03-19 09:56:25,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/c4143803a85d47e7b16a94b8e941bcbf as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/c4143803a85d47e7b16a94b8e941bcbf 2023-03-19 09:56:25,427 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/c4143803a85d47e7b16a94b8e941bcbf, entries=150, sequenceid=68, filesize=11.6 K 2023-03-19 09:56:25,428 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for a391bedbca383dedc4bc8fc94dd92901 in 84ms, sequenceid=68, compaction requested=true 2023-03-19 09:56:25,428 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:25,428 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:25,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:25,428 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:25,429 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:25,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4fd985[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:25,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:25,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,436 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:25,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:25,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:25,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:25,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:25,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,438 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:25,440 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:25,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:25,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:25,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:25,441 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:25,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,442 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,443 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,444 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,448 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,448 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,448 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,448 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,449 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,449 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,449 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,448 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:25,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:25,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,449 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,449 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:25,449 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:25,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,549 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:25,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:25,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:25,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:25,552 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:25,552 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,552 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:25,552 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:25,553 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,553 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:25,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:25,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:25,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:25,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:25,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:25,558 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:25,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39efabe4[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:25,559 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:25,560 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,561 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,561 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:25,564 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:25,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:25,567 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:25,567 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:25,567 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:25,567 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:25,567 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:25,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,569 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,569 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,574 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,576 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:25,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,576 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,577 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:25,577 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:25,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,677 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:25,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:25,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:25,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:25,680 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:25,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:25,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:25,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,681 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,695 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:25,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:25,697 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,697 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:25,697 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:25,697 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,698 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:25,698 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:25,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f44f5b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:25,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:25,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,702 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,702 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:25,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:25,702 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:25,702 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:25,704 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:25,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:25,706 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:25,706 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:25,706 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:25,706 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:25,706 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:25,708 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,709 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,710 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:25,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:25,713 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,713 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:25,713 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,713 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:25,814 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:25,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:25,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:25,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:25,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:25,818 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:25,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:25,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:25,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,823 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,823 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,823 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,823 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:25,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:25,825 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:25,825 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,825 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:25,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:25,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:25,826 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,826 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:25,827 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:25,827 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:25,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ded0ab7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:25,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:25,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,832 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,832 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:25,832 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:25,832 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:25,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:25,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:25,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:25,839 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:25,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:25,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:25,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:25,839 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:25,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,842 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,843 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,846 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:25,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,846 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:25,846 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,846 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,946 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:25,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:25,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:25,952 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:25,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:25,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:25,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,954 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,954 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,955 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:25,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:25,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:25,957 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,958 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:25,958 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:25,958 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:25,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,961 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:25,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497a737e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:25,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:25,962 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,964 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:25,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:25,965 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,965 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,966 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,966 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:25,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:25,968 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:25,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:25,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:25,968 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:25,969 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:25,970 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,971 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:25,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,971 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:25,973 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:25,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:25,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:25,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:25,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:25,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:25,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:25,976 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:25,976 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:25,977 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:25,977 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:25,977 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:25,977 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:25,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:25,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:26,077 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,080 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,080 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,081 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,081 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,081 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,081 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,082 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,083 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,083 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,085 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,085 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,088 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,088 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d0e413e[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,093 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,094 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,094 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,098 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,098 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,100 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,104 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:26,104 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,104 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,104 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:26,199 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:56:26,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:26,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:26,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:26,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:26,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:26,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:26,205 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,211 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,211 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,212 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,213 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,213 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,214 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,215 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,215 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,215 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:26,215 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,215 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219786210, max timestamp=1679219786215Num uniques 859; 2023-03-19 09:56:26,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219786210, max timestamp=1679219786215Num uniques 859; 2023-03-19 09:56:26,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:26,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219786210, max timestamp=1679219786215Num uniques 859; 2023-03-19 09:56:26,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:26,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:26,218 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60b4953[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,220 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,224 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,225 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,226 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,228 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,228 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,229 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,230 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,231 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,238 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,239 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,239 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,238 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,239 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219846252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219846255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219846256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219846257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219846257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,339 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,342 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,342 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,343 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,345 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:26,345 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33864397[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,349 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,350 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,350 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,350 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,350 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,352 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,354 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,354 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,355 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,355 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,356 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219846358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,360 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,361 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219846359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219846361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,363 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,363 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,363 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,363 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219846362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219846363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,464 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,467 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,467 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,468 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,468 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,469 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,469 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,470 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,470 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,470 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:26,471 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,473 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29fd271a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,475 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,476 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,476 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,477 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,478 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,480 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,480 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,480 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,482 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,484 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,487 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,487 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,487 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,487 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219846559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219846563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219846564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219846567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219846567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,587 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,590 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,592 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,593 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,593 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,593 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:26,593 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,595 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@560584aa[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,596 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,597 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,598 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,598 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,602 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,602 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,603 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,605 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,605 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,605 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,606 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,606 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,606 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,607 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,608 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,612 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,612 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,612 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,612 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,614 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,615 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,615 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:26,616 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303197bbe0189505d469ab4d92dc15d85d021 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197bbe0189505d469ab4d92dc15d85d021 2023-03-19 09:56:26,620 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/7e63a3e95dc14c98b91d0ff8741dfbee 2023-03-19 09:56:26,715 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,719 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,719 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,720 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,720 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,722 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,722 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,723 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,724 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,724 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,724 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:26,724 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,724 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,725 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,725 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,726 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a181ee8[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,727 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,728 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,728 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,730 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,732 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,732 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,732 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,734 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,735 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,738 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:26,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,738 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,739 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,739 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,739 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,739 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,739 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,739 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,839 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,842 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,842 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,842 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,842 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,843 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,843 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,844 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,845 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,845 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,845 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,846 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,846 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,846 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:26,846 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,846 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,847 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,847 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,848 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b837ec7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,849 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,849 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,851 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,851 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,851 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,855 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,855 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,856 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,856 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,856 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,857 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,858 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,860 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,860 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,861 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,861 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,861 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219846863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219846866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219846867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219846871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:26,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219846871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,961 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:26,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:26,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:26,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:26,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:26,964 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:26,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:26,964 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:26,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,965 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,966 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,966 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,967 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:26,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:26,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:26,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:26,968 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:26,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:26,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:26,969 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:26,970 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:26,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b064d5a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:26,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:26,972 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,973 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:26,973 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,973 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:26,974 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:26,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,974 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:26,976 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:26,977 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:26,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,980 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:26,980 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:26,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:26,981 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:26,985 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:26,986 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:26,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:26,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:26,986 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:26,986 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:26,986 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/29f41eba07a34617a0bb6596ef2aa911 2023-03-19 09:56:27,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/2dd953283ca14521a15aeb074f774b54 2023-03-19 09:56:27,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/7e63a3e95dc14c98b91d0ff8741dfbee as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7e63a3e95dc14c98b91d0ff8741dfbee 2023-03-19 09:56:27,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7e63a3e95dc14c98b91d0ff8741dfbee, entries=100, sequenceid=83, filesize=18.4 K 2023-03-19 09:56:27,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/29f41eba07a34617a0bb6596ef2aa911 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/29f41eba07a34617a0bb6596ef2aa911 2023-03-19 09:56:27,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/29f41eba07a34617a0bb6596ef2aa911, entries=100, sequenceid=83, filesize=9.3 K 2023-03-19 09:56:27,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/2dd953283ca14521a15aeb074f774b54 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/2dd953283ca14521a15aeb074f774b54 2023-03-19 09:56:27,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/2dd953283ca14521a15aeb074f774b54, entries=100, sequenceid=83, filesize=9.3 K 2023-03-19 09:56:27,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a391bedbca383dedc4bc8fc94dd92901 in 852ms, sequenceid=83, compaction requested=true 2023-03-19 09:56:27,051 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:27,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:27,052 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:56:27,052 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:56:27,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:27,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:27,053 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 121782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:56:27,053 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:56:27,053 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:27,053 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:27,053 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,053 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,053 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/49f819207a954b3e819c5a2aecae0cc7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/93feb0764c4a4256a2587e18bd7865c0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5940edeb7f764922bd26b689e3de117c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/3f0906d6af6c4bac8d9ee4330237c2d3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7e63a3e95dc14c98b91d0ff8741dfbee] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=118.9 K 2023-03-19 09:56:27,053 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/14d4323f727c4be5989b896ec5b282df, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/438ad338bc1346aabfb2fe28af0cd870, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6c1f3d1782b04947944803a78db7bfeb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/60922ab72b6c4917b5063009c15c225c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/29f41eba07a34617a0bb6596ef2aa911] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=55.7 K 2023-03-19 09:56:27,054 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 49f819207a954b3e819c5a2aecae0cc7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1679219782281 2023-03-19 09:56:27,054 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 14d4323f727c4be5989b896ec5b282df, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1679219782281 2023-03-19 09:56:27,054 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 93feb0764c4a4256a2587e18bd7865c0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679219782307 2023-03-19 09:56:27,054 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 438ad338bc1346aabfb2fe28af0cd870, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679219782307 2023-03-19 09:56:27,054 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5940edeb7f764922bd26b689e3de117c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679219783435 2023-03-19 09:56:27,054 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6c1f3d1782b04947944803a78db7bfeb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679219783435 2023-03-19 09:56:27,054 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 3f0906d6af6c4bac8d9ee4330237c2d3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679219784058 2023-03-19 09:56:27,055 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 60922ab72b6c4917b5063009c15c225c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679219784058 2023-03-19 09:56:27,055 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7e63a3e95dc14c98b91d0ff8741dfbee, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219786195 2023-03-19 09:56:27,055 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 29f41eba07a34617a0bb6596ef2aa911, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219786195 2023-03-19 09:56:27,063 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#535 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:27,064 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#534 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:27,081 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/161c0cdf5921418fbab3daf64660262b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/161c0cdf5921418fbab3daf64660262b 2023-03-19 09:56:27,085 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 161c0cdf5921418fbab3daf64660262b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:27,085 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:27,085 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=11, startTime=1679219787052; duration=0sec 2023-03-19 09:56:27,086 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:27,086 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:56:27,086 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:27,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:27,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:56:27,087 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:27,087 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,087 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fd16037091d446ff9a15f4a82219210c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/6b70e918ac7245c19f86a5a69a4d70ad, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/becc227127f24f6b825701f8bbca7053, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/c4143803a85d47e7b16a94b8e941bcbf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/2dd953283ca14521a15aeb074f774b54] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=55.7 K 2023-03-19 09:56:27,088 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fd16037091d446ff9a15f4a82219210c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1679219782281 2023-03-19 09:56:27,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6b70e918ac7245c19f86a5a69a4d70ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679219782307 2023-03-19 09:56:27,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:27,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting becc227127f24f6b825701f8bbca7053, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679219783435 2023-03-19 09:56:27,089 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c4143803a85d47e7b16a94b8e941bcbf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679219784058 2023-03-19 09:56:27,090 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 2dd953283ca14521a15aeb074f774b54, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219786195 2023-03-19 09:56:27,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:27,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:27,090 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:27,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:27,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:27,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:27,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,092 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:27,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,092 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:27,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:27,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:27,093 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,093 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:27,093 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:27,093 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:56:27,094 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:27,094 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:27,094 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:27,094 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:27,094 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:27,094 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:27,097 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#536 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:27,114 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319493516085d914f0b9765047e709f0cfe to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319493516085d914f0b9765047e709f0cfe 2023-03-19 09:56:27,118 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=121, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/592d75135cc64487a1e7c4bd3423c6bf 2023-03-19 09:56:27,125 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/c62a41dcde59438d85fdc0dc692f11ce 2023-03-19 09:56:27,137 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/eafc34d011914c919ab0a7652eba7556 2023-03-19 09:56:27,140 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/592d75135cc64487a1e7c4bd3423c6bf as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/592d75135cc64487a1e7c4bd3423c6bf 2023-03-19 09:56:27,143 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/592d75135cc64487a1e7c4bd3423c6bf, entries=150, sequenceid=121, filesize=25.1 K 2023-03-19 09:56:27,144 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/c62a41dcde59438d85fdc0dc692f11ce as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/c62a41dcde59438d85fdc0dc692f11ce 2023-03-19 09:56:27,147 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/c62a41dcde59438d85fdc0dc692f11ce, entries=150, sequenceid=121, filesize=11.6 K 2023-03-19 09:56:27,148 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/eafc34d011914c919ab0a7652eba7556 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/eafc34d011914c919ab0a7652eba7556 2023-03-19 09:56:27,151 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/eafc34d011914c919ab0a7652eba7556, entries=150, sequenceid=121, filesize=11.6 K 2023-03-19 09:56:27,151 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for a391bedbca383dedc4bc8fc94dd92901 in 58ms, sequenceid=121, compaction requested=false 2023-03-19 09:56:27,151 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:27,151 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:27,153 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@315cb044[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:27,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:27,155 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,157 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:27,157 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,157 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:27,158 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:27,158 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:27,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:27,161 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:27,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:27,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:27,161 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:27,161 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:27,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,163 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:27,163 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,165 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:27,167 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:27,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:27,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,167 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,167 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:27,167 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:27,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:27,267 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:27,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:27,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:27,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:27,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:27,271 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:27,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:27,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:27,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:27,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,272 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,274 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:27,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:27,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:27,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:27,276 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,276 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:27,276 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:27,277 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,278 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7907d9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:27,280 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:27,280 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,281 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:27,281 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:27,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,281 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:27,282 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:27,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:27,282 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:27,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:27,285 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:27,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:27,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:27,285 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:27,285 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:27,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:27,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:27,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:27,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:27,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:27,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:27,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:27,291 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:27,291 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:27,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:27,291 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:27,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:27,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:27,403 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:27,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:27,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:27,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:27,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:27,408 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:27,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:27,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:27,408 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:27,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,409 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:27,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:27,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,410 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:27,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:27,412 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:27,412 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:27,412 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:27,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:27,442 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303198d4c87019b544a7dadacc86b3428be3a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198d4c87019b544a7dadacc86b3428be3a 2023-03-19 09:56:27,447 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/95260dcb7c4846a18c6d842f0dd909ee 2023-03-19 09:56:27,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219787437, max timestamp=1679219787452Num uniques 859; 2023-03-19 09:56:27,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:56:27,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219787437, max timestamp=1679219787452Num uniques 859; 2023-03-19 09:56:27,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679219787437, max timestamp=1679219787452Num uniques 859; 2023-03-19 09:56:27,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:56:27,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-19 09:56:27,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219847468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219787462, max timestamp=1679219787472Num uniques 1068; 2023-03-19 09:56:27,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-19 09:56:27,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219787462, max timestamp=1679219787472Num uniques 1068; 2023-03-19 09:56:27,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-19 09:56:27,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679219787462, max timestamp=1679219787472Num uniques 1068; 2023-03-19 09:56:27,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-19 09:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679219787487, max timestamp=1679219787493Num uniques 1127; 2023-03-19 09:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679219787487, max timestamp=1679219787493Num uniques 1127; 2023-03-19 09:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679219787487, max timestamp=1679219787493Num uniques 1127; 2023-03-19 09:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-19 09:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-19 09:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-19 09:56:27,509 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/fcb3f01aacac4eeda2e5a1b0fd83d1ba as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fcb3f01aacac4eeda2e5a1b0fd83d1ba 2023-03-19 09:56:27,513 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into fcb3f01aacac4eeda2e5a1b0fd83d1ba(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:27,513 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:27,513 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=11, startTime=1679219787052; duration=0sec 2023-03-19 09:56:27,513 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:27,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219847509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219847512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219847514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219847514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219847575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219847614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219847617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219847620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219847620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219847776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219847818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219847820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219847823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:27,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219847824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:27,857 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/1788ac20605f4828afa47726aa3f4614 2023-03-19 09:56:28,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219848079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219848121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219848124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219848128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219848128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,267 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/a13102c9479f44448eb02e0674e27416 2023-03-19 09:56:28,271 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/95260dcb7c4846a18c6d842f0dd909ee as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/95260dcb7c4846a18c6d842f0dd909ee 2023-03-19 09:56:28,274 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/95260dcb7c4846a18c6d842f0dd909ee, entries=250, sequenceid=130, filesize=38.8 K 2023-03-19 09:56:28,275 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/1788ac20605f4828afa47726aa3f4614 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1788ac20605f4828afa47726aa3f4614 2023-03-19 09:56:28,278 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1788ac20605f4828afa47726aa3f4614, entries=100, sequenceid=130, filesize=9.4 K 2023-03-19 09:56:28,279 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/a13102c9479f44448eb02e0674e27416 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a13102c9479f44448eb02e0674e27416 2023-03-19 09:56:28,284 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a13102c9479f44448eb02e0674e27416, entries=100, sequenceid=130, filesize=9.4 K 2023-03-19 09:56:28,285 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for a391bedbca383dedc4bc8fc94dd92901 in 873ms, sequenceid=130, compaction requested=true 2023-03-19 09:56:28,285 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:28,285 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:28,285 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:28,285 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:28,285 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,285 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:28,285 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:28,287 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,287 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:28,287 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,287 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:28,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:28,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1c2b37[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:28,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:28,289 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,290 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/9acbe182f9c343639ca4402b1889ff45 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9acbe182f9c343639ca4402b1889ff45 2023-03-19 09:56:28,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:28,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:28,292 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,294 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into 9acbe182f9c343639ca4402b1889ff45(size=25.3 K), total size for store is 89.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-19 09:56:28,294 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:28,294 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=11, startTime=1679219787052; duration=1sec 2023-03-19 09:56:28,294 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:28,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:28,295 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:28,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:28,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:28,295 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:28,295 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:28,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:28,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,297 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:28,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:28,298 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:28,299 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:28,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:28,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:28,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:28,303 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:28,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:28,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:28,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:28,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:28,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:28,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:28,304 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:28,304 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:28,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:28,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:28,304 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,404 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:28,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:28,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:28,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:28,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:28,414 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:28,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:28,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:28,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:28,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,415 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:28,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,416 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:28,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:28,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:28,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:28,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:28,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:28,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:28,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:28,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:28,423 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=154.31 KB heapSize=429 KB 2023-03-19 09:56:28,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:28,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:28,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:28,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:28,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:28,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:28,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:28,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:28,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219848604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219848625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219848630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219848635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219848636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219848708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:28,847 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303196ce5cc152a7249389739e3bcbf745f64 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196ce5cc152a7249389739e3bcbf745f64 2023-03-19 09:56:28,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:28,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219848911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:29,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:29,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219849216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:29,262 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/33e9e4dee7404f0f8d32e917adc75f85 2023-03-19 09:56:29,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219849634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:29,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219849642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219849643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:29,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:29,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219849644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:29,690 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/b773d37920074c9bbd9d355b4c389091 2023-03-19 09:56:29,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36762 deadline: 1679219849723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,106 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/3531f1c7e87545029d43fccc67fcca46 2023-03-19 09:56:30,109 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/33e9e4dee7404f0f8d32e917adc75f85 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/33e9e4dee7404f0f8d32e917adc75f85 2023-03-19 09:56:30,112 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/33e9e4dee7404f0f8d32e917adc75f85, entries=150, sequenceid=203, filesize=25.3 K 2023-03-19 09:56:30,113 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/b773d37920074c9bbd9d355b4c389091 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b773d37920074c9bbd9d355b4c389091 2023-03-19 09:56:30,116 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b773d37920074c9bbd9d355b4c389091, entries=150, sequenceid=203, filesize=11.8 K 2023-03-19 09:56:30,117 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/3531f1c7e87545029d43fccc67fcca46 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3531f1c7e87545029d43fccc67fcca46 2023-03-19 09:56:30,120 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3531f1c7e87545029d43fccc67fcca46, entries=150, sequenceid=203, filesize=11.8 K 2023-03-19 09:56:30,121 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~428.95 KB/439248, currentSize=33.54 KB/34350 for a391bedbca383dedc4bc8fc94dd92901 in 1697ms, sequenceid=203, compaction requested=true 2023-03-19 09:56:30,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:30,121 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:30,122 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,122 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1209c793[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:30,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:30,126 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,127 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,127 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:30,127 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:30,127 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:30,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,127 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:30,131 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:30,134 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:30,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:30,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:30,134 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:30,134 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:30,136 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,136 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,136 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,137 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,138 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,141 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:30,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:30,141 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,142 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,142 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:30,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,142 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:30,242 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:30,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:30,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:30,245 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:30,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,246 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,247 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,247 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,247 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,247 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:30,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:30,248 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,248 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:30,249 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-19 09:56:30,249 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:30,249 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:30,249 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:30,249 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:30,249 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:30,249 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:30,259 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319a53e5c942ef54a3aad08f35699194a92 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a53e5c942ef54a3aad08f35699194a92 2023-03-19 09:56:30,264 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=211, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/5572e54aa7914819a50ab32afad17041 2023-03-19 09:56:30,272 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/71dfd963c2c048edb8e13fe618101e9e 2023-03-19 09:56:30,281 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/965513e5d0c342d5a833ef73080e4262 2023-03-19 09:56:30,285 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/5572e54aa7914819a50ab32afad17041 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5572e54aa7914819a50ab32afad17041 2023-03-19 09:56:30,289 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5572e54aa7914819a50ab32afad17041, entries=150, sequenceid=211, filesize=25.3 K 2023-03-19 09:56:30,290 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/71dfd963c2c048edb8e13fe618101e9e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/71dfd963c2c048edb8e13fe618101e9e 2023-03-19 09:56:30,293 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/71dfd963c2c048edb8e13fe618101e9e, entries=150, sequenceid=211, filesize=11.8 K 2023-03-19 09:56:30,293 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/965513e5d0c342d5a833ef73080e4262 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/965513e5d0c342d5a833ef73080e4262 2023-03-19 09:56:30,296 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/965513e5d0c342d5a833ef73080e4262, entries=150, sequenceid=211, filesize=11.8 K 2023-03-19 09:56:30,297 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for a391bedbca383dedc4bc8fc94dd92901 in 48ms, sequenceid=211, compaction requested=true 2023-03-19 09:56:30,297 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:30,297 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,300 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46039b06[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:30,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:30,302 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,303 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:30,304 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,304 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:30,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:30,304 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:30,305 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:30,308 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:30,308 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:30,308 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:30,308 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:30,308 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:30,310 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,310 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,310 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,311 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,314 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,315 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:30,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,315 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:30,315 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:30,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:30,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,415 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:30,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:30,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:30,419 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:30,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:30,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:30,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,419 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,420 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,420 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,421 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:30,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:30,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,422 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:30,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:30,423 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:30,424 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,425 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75771bd0[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:30,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:30,426 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,427 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:30,427 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:30,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:30,434 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:30,438 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:30,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:30,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:30,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:30,438 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:30,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,439 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,441 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:30,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:30,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:30,443 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:30,443 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:30,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,444 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,443 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,544 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:30,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:30,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:30,548 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:30,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:30,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:30,549 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,549 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,554 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,554 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:30,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:30,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:30,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,556 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:30,557 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:30,557 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:30,558 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,559 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:30,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f38c3da[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:30,561 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:30,561 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,562 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,562 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:30,562 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:30,563 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:30,563 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:30,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:30,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,564 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:30,564 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:30,582 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:30,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:30,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:30,583 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:30,583 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:30,585 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,585 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,585 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,586 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,587 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,589 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:30,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:30,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,590 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,590 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,590 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,589 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,590 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,590 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,590 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:30,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,690 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:30,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:30,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:30,694 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:30,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:30,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:30,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,694 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,695 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,695 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:30,699 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:30,699 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:30,699 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:30,699 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:30,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:30,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d1a6f1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:30,703 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:30,703 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:30,705 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:30,705 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:30,705 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:30,707 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,706 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,707 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:30,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:30,709 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:30,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:30,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:30,709 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:30,709 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:30,711 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,711 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,711 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,711 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,713 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,715 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:30,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:30,715 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,716 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:30,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,716 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:30,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,716 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:30,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:56:30,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:30,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:30,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:30,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:30,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:30,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:30,796 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303192808b5c6f3a6402883c322c6cff7cc72 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303192808b5c6f3a6402883c322c6cff7cc72 2023-03-19 09:56:30,816 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:30,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:30,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:30,840 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/1a4947223de24711987dcafb8ee43efa 2023-03-19 09:56:30,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:30,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:30,852 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:30,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:30,852 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:30,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,855 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,855 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,856 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:30,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:30,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:30,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:30,876 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,876 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:30,877 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:30,877 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219790857, max timestamp=1679219790879Num uniques 859; 2023-03-19 09:56:30,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:30,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219790857, max timestamp=1679219790879Num uniques 859; 2023-03-19 09:56:30,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219790857, max timestamp=1679219790879Num uniques 859; 2023-03-19 09:56:30,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:30,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:30,905 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,905 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:30,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7950c226[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:30,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:30,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,911 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,911 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:30,911 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:30,911 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:30,911 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,911 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:30,913 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,914 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/39035599cf6c468e9f5c69dd3e49d84a 2023-03-19 09:56:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:30,918 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:30,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:30,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:30,918 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:30,918 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:30,923 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,923 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:30,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,923 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,924 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:30,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:30,932 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:30,932 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:30,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:30,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:30,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:30,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:30,933 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:30,933 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:30,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:30,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:30,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:30,933 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:30,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:30,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b8d84b19a94e4a3ca5e0cb90b44ae264 2023-03-19 09:56:30,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/1a4947223de24711987dcafb8ee43efa as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1a4947223de24711987dcafb8ee43efa 2023-03-19 09:56:30,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1a4947223de24711987dcafb8ee43efa, entries=300, sequenceid=224, filesize=45.8 K 2023-03-19 09:56:30,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/39035599cf6c468e9f5c69dd3e49d84a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/39035599cf6c468e9f5c69dd3e49d84a 2023-03-19 09:56:30,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/39035599cf6c468e9f5c69dd3e49d84a, entries=100, sequenceid=224, filesize=9.4 K 2023-03-19 09:56:30,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b8d84b19a94e4a3ca5e0cb90b44ae264 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b8d84b19a94e4a3ca5e0cb90b44ae264 2023-03-19 09:56:30,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b8d84b19a94e4a3ca5e0cb90b44ae264, entries=100, sequenceid=224, filesize=9.4 K 2023-03-19 09:56:30,981 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=127.47 KB/130530 for a391bedbca383dedc4bc8fc94dd92901 in 202ms, sequenceid=224, compaction requested=true 2023-03-19 09:56:30,981 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:30,981 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:30,981 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-19 09:56:30,981 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-19 09:56:30,981 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:30,981 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:30,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:30,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:56:30,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:30,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:30,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:30,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:30,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:30,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:30,985 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67250 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-19 09:56:30,985 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:30,985 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 190048 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-19 09:56:30,985 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,985 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:30,985 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/161c0cdf5921418fbab3daf64660262b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/c62a41dcde59438d85fdc0dc692f11ce, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1788ac20605f4828afa47726aa3f4614, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b773d37920074c9bbd9d355b4c389091, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/71dfd963c2c048edb8e13fe618101e9e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/39035599cf6c468e9f5c69dd3e49d84a] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=65.7 K 2023-03-19 09:56:30,985 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:30,985 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9acbe182f9c343639ca4402b1889ff45, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/592d75135cc64487a1e7c4bd3423c6bf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/95260dcb7c4846a18c6d842f0dd909ee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/33e9e4dee7404f0f8d32e917adc75f85, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5572e54aa7914819a50ab32afad17041, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1a4947223de24711987dcafb8ee43efa] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=185.6 K 2023-03-19 09:56:30,986 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 161c0cdf5921418fbab3daf64660262b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219784059 2023-03-19 09:56:30,986 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9acbe182f9c343639ca4402b1889ff45, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219784059 2023-03-19 09:56:30,986 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting c62a41dcde59438d85fdc0dc692f11ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679219786251 2023-03-19 09:56:30,986 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 592d75135cc64487a1e7c4bd3423c6bf, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679219786251 2023-03-19 09:56:30,987 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1788ac20605f4828afa47726aa3f4614, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1679219787407 2023-03-19 09:56:30,987 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 95260dcb7c4846a18c6d842f0dd909ee, keycount=250, bloomtype=ROW, size=38.8 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1679219787405 2023-03-19 09:56:30,987 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b773d37920074c9bbd9d355b4c389091, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679219787508 2023-03-19 09:56:30,987 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 71dfd963c2c048edb8e13fe618101e9e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1679219788587 2023-03-19 09:56:30,987 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 33e9e4dee7404f0f8d32e917adc75f85, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679219787508 2023-03-19 09:56:30,988 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 39035599cf6c468e9f5c69dd3e49d84a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679219790769 2023-03-19 09:56:30,988 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5572e54aa7914819a50ab32afad17041, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1679219788587 2023-03-19 09:56:30,990 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1a4947223de24711987dcafb8ee43efa, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679219790744 2023-03-19 09:56:31,013 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#554 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:31,019 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319daf794f3aaa843b499a00e36d7df6549 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319daf794f3aaa843b499a00e36d7df6549 2023-03-19 09:56:31,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219851022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,033 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:31,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:31,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:31,039 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#553 average throughput is 0.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:31,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:31,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:31,048 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:31,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:31,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:31,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:31,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,050 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:31,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:31,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:31,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:31,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:31,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:31,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:31,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:31,062 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:31,067 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/11e8cbdb1cc940d0990690d49114338c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/11e8cbdb1cc940d0990690d49114338c 2023-03-19 09:56:31,069 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:31,069 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:31,069 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:31,069 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:31,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:31,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,072 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 11e8cbdb1cc940d0990690d49114338c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:31,072 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:31,072 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=10, startTime=1679219790981; duration=0sec 2023-03-19 09:56:31,072 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:31,072 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-19 09:56:31,074 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67250 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-19 09:56:31,074 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:31,074 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:31,074 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fcb3f01aacac4eeda2e5a1b0fd83d1ba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/eafc34d011914c919ab0a7652eba7556, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a13102c9479f44448eb02e0674e27416, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3531f1c7e87545029d43fccc67fcca46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/965513e5d0c342d5a833ef73080e4262, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b8d84b19a94e4a3ca5e0cb90b44ae264] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=65.7 K 2023-03-19 09:56:31,075 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fcb3f01aacac4eeda2e5a1b0fd83d1ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679219784059 2023-03-19 09:56:31,075 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting eafc34d011914c919ab0a7652eba7556, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679219786251 2023-03-19 09:56:31,079 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a13102c9479f44448eb02e0674e27416, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1679219787407 2023-03-19 09:56:31,080 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 3531f1c7e87545029d43fccc67fcca46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679219787508 2023-03-19 09:56:31,080 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 965513e5d0c342d5a833ef73080e4262, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1679219788587 2023-03-19 09:56:31,081 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b8d84b19a94e4a3ca5e0cb90b44ae264, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679219790769 2023-03-19 09:56:31,084 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=261, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/21a6415becf243d7a01458396db37e61 2023-03-19 09:56:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,087 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:31,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:31,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3434ef47[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:31,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:31,090 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,096 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:31,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,097 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:31,098 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:31,102 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:31,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:31,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:31,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:31,102 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:31,106 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#556 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:31,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:31,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:31,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,110 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:31,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,114 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:31,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:31,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:31,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:31,117 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:31,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:31,117 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:31,117 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:31,118 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:31,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:31,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219851127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/82ed10b1018443bfae45525c4853ced2 2023-03-19 09:56:31,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/daea4e44f5954268950d6c0e2842ecd3 2023-03-19 09:56:31,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/21a6415becf243d7a01458396db37e61 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/21a6415becf243d7a01458396db37e61 2023-03-19 09:56:31,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/21a6415becf243d7a01458396db37e61, entries=200, sequenceid=261, filesize=32.3 K 2023-03-19 09:56:31,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/82ed10b1018443bfae45525c4853ced2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/82ed10b1018443bfae45525c4853ced2 2023-03-19 09:56:31,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/82ed10b1018443bfae45525c4853ced2, entries=150, sequenceid=261, filesize=11.9 K 2023-03-19 09:56:31,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/daea4e44f5954268950d6c0e2842ecd3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/daea4e44f5954268950d6c0e2842ecd3 2023-03-19 09:56:31,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/daea4e44f5954268950d6c0e2842ecd3, entries=150, sequenceid=261, filesize=11.9 K 2023-03-19 09:56:31,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for a391bedbca383dedc4bc8fc94dd92901 in 188ms, sequenceid=261, compaction requested=false 2023-03-19 09:56:31,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:31,218 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:31,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:31,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:31,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:31,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:31,223 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:31,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:31,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:31,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:31,223 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,224 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,226 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:31,226 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,226 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,226 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:31,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:31,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:31,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:31,233 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:31,233 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:31,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:31,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:31,233 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:56:31,234 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:31,235 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:31,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:31,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:31,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:31,236 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:31,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:31,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:31,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219791388, max timestamp=1679219791399Num uniques 859; 2023-03-19 09:56:31,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:31,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219791388, max timestamp=1679219791399Num uniques 859; 2023-03-19 09:56:31,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219791388, max timestamp=1679219791399Num uniques 859; 2023-03-19 09:56:31,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:31,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:31,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219851454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,546 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/988b7be44fd2405aa3238e120342b0a1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/988b7be44fd2405aa3238e120342b0a1 2023-03-19 09:56:31,550 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 988b7be44fd2405aa3238e120342b0a1(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:31,550 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:31,550 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=10, startTime=1679219790981; duration=0sec 2023-03-19 09:56:31,550 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:31,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219851557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219851641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,645 DEBUG [Thread-3953] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:31,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219851645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,652 DEBUG [Thread-3959] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:31,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219851649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,654 DEBUG [Thread-3957] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:31,660 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031939dc9d7a05c04ada88810b383d94a275 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031939dc9d7a05c04ada88810b383d94a275 2023-03-19 09:56:31,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219851666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,669 DEBUG [Thread-3951] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:31,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:31,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219851761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:31,934 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/0774fe9157c64e7983a63f492c44cfd3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0774fe9157c64e7983a63f492c44cfd3 2023-03-19 09:56:31,937 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into 0774fe9157c64e7983a63f492c44cfd3(size=25.7 K), total size for store is 58.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:31,937 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:31,937 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=10, startTime=1679219790981; duration=0sec 2023-03-19 09:56:31,938 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:32,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:32,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219852065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,083 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/5d507181d5a44813a752f5b48ab4c4b5 2023-03-19 09:56:32,091 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/55618be14551471ba1fa301605961b0c 2023-03-19 09:56:32,500 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/cc007ca695224e189968bd2c9a8147b1 2023-03-19 09:56:32,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/5d507181d5a44813a752f5b48ab4c4b5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5d507181d5a44813a752f5b48ab4c4b5 2023-03-19 09:56:32,507 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5d507181d5a44813a752f5b48ab4c4b5, entries=100, sequenceid=272, filesize=18.6 K 2023-03-19 09:56:32,507 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/55618be14551471ba1fa301605961b0c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55618be14551471ba1fa301605961b0c 2023-03-19 09:56:32,510 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55618be14551471ba1fa301605961b0c, entries=100, sequenceid=272, filesize=9.5 K 2023-03-19 09:56:32,511 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/cc007ca695224e189968bd2c9a8147b1 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cc007ca695224e189968bd2c9a8147b1 2023-03-19 09:56:32,514 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cc007ca695224e189968bd2c9a8147b1, entries=100, sequenceid=272, filesize=9.5 K 2023-03-19 09:56:32,514 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a391bedbca383dedc4bc8fc94dd92901 in 1281ms, sequenceid=272, compaction requested=true 2023-03-19 09:56:32,515 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:32,515 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:32,516 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,517 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,517 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:32,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dabdc08[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:32,518 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:32,518 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,520 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:32,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:32,521 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:32,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:32,523 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:32,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:32,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:32,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:32,524 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:32,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,525 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,525 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,527 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:32,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:32,529 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,529 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:32,529 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,530 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:32,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:32,574 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-19 09:56:32,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:32,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:56:32,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:32,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:56:32,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:32,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-19 09:56:32,602 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319f7c83ae8742741e4829187ec92c4612d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319f7c83ae8742741e4829187ec92c4612d 2023-03-19 09:56:32,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:32,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219852614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,630 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:32,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:32,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:32,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:32,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:32,640 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:32,640 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:32,640 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:32,640 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:32,641 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,641 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,643 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:32,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:32,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,643 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:32,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:32,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:32,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:32,644 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:32,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:32,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:32,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:32,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:32,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:32,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:32,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:32,645 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:32,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:32,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:32,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:32,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:32,647 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,647 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:32,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23630562[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:32,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:32,650 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,651 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:32,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:32,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:32,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:32,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:32,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,653 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:32,653 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:32,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:32,657 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:32,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:32,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:32,657 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:32,657 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:32,659 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,659 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,659 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,660 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,662 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,664 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,664 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,664 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,664 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,665 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:32,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:32,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:32,665 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:32,665 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:32,665 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:32,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:32,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219852718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,765 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:32,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:32,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:32,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:32,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:32,770 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:32,770 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:32,770 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:32,770 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:32,771 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,771 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,772 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:32,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:32,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:32,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,772 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:32,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:32,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:32,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:32,774 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:32,774 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:32,774 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:32,774 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:32,774 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:32,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,776 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c5da2da[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:32,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:32,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,785 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:32,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,786 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:32,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:32,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,793 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:32,793 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:32,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:32,796 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:32,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:32,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:32,797 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:32,797 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:32,798 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,798 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,798 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,799 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,801 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:32,804 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:32,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,804 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,804 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:32,805 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:32,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,905 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:32,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:32,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:32,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:32,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:32,908 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:32,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:32,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:32,908 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:32,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,909 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,910 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:32,910 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:32,910 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,910 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:32,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:32,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:32,916 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:32,916 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:32,916 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:32,916 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:32,918 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:32,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f8020c1[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:32,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:32,920 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,921 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:32,921 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:32,921 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:32,921 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:32,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:32,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219852921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,926 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:32,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,926 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:32,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:32,929 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:32,929 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:32,929 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:32,929 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:32,929 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:32,930 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,930 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,931 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:32,931 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:32,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,932 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:32,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:32,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:32,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:32,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:32,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:32,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:32,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:32,935 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:32,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:32,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:32,935 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:32,935 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:32,935 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,936 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:32,936 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,011 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/2ac845969a214e8482c44cbde0fa4c61 2023-03-19 09:56:33,035 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:33,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:33,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:33,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:33,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:33,043 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:33,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:33,043 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:33,044 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,044 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,045 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,045 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,046 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,046 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:33,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:33,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:33,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:33,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:33,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:33,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:33,056 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,056 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:33,056 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:33,056 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:33,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:33,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:33,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:33,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/45e7cb265815401eb7ed03ee244530e3 2023-03-19 09:56:33,059 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c18fea5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:33,063 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:33,063 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,066 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:33,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:33,066 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:33,067 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,067 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:33,067 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,067 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:33,067 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,067 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:33,070 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:33,070 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:33,070 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:33,070 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:33,070 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:33,072 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,072 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,072 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,074 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,078 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,081 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:33,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:33,081 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,082 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,082 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:33,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,082 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:33,182 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:33,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:33,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:33,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:33,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:33,186 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:33,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:33,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:33,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,186 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,188 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,188 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,193 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,193 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:33,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:33,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:33,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:33,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:33,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:33,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:33,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:33,196 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:33,197 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,197 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:33,197 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:33,197 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:33,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:33,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:33,197 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:33,198 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,198 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,199 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:33,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ebb051[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:33,200 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:33,200 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,201 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:33,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:33,201 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,202 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:33,201 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:33,204 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:33,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,204 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:33,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:33,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:33,207 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:33,207 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:33,207 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:33,207 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:33,207 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:33,209 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,209 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,209 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,209 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,210 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,217 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:33,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:33,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:33,217 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:33,217 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:33,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:33,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219853226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,318 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:33,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:33,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:33,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:33,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:33,321 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:33,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:33,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:33,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,322 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:33,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:33,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:33,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:33,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:33,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:33,326 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:33,327 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,327 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:33,327 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:33,327 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,327 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:33,327 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:33,327 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,327 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:33,327 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:33,329 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,329 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,330 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,330 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:33,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16918d04[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:33,333 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:33,333 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:33,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:33,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:33,335 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,335 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:33,336 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:33,340 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:33,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:33,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:33,340 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:33,340 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:33,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,343 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,346 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:33,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:33,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:33,348 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,349 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:33,349 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:33,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,449 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:33,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:33,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:33,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:33,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:33,456 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:33,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:33,456 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:33,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,457 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,458 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,458 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,459 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:33,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:33,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:33,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:33,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:33,460 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:33,460 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,461 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:33,461 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:33,461 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:33,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:33,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:33,461 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:33,462 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,462 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,462 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,463 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:33,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5859213f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:33,465 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:33,465 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,470 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:33,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,470 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:33,471 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:33,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,471 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:33,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:33,474 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:33,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:33,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:33,474 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:33,474 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:33,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,475 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/4f3326e9b69a4dbab70b0f1b92bbe189 2023-03-19 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:33,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,482 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:33,483 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/2ac845969a214e8482c44cbde0fa4c61 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/2ac845969a214e8482c44cbde0fa4c61 2023-03-19 09:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:33,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,485 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:33,485 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:33,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:33,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:33,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:33,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,486 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:33,486 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:33,486 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:33,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:33,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/2ac845969a214e8482c44cbde0fa4c61, entries=150, sequenceid=312, filesize=25.4 K 2023-03-19 09:56:33,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/45e7cb265815401eb7ed03ee244530e3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/45e7cb265815401eb7ed03ee244530e3 2023-03-19 09:56:33,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/45e7cb265815401eb7ed03ee244530e3, entries=150, sequenceid=312, filesize=11.9 K 2023-03-19 09:56:33,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/4f3326e9b69a4dbab70b0f1b92bbe189 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/4f3326e9b69a4dbab70b0f1b92bbe189 2023-03-19 09:56:33,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/4f3326e9b69a4dbab70b0f1b92bbe189, entries=150, sequenceid=312, filesize=11.9 K 2023-03-19 09:56:33,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for a391bedbca383dedc4bc8fc94dd92901 in 921ms, sequenceid=312, compaction requested=true 2023-03-19 09:56:33,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:33,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:33,496 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:33,496 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:33,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:33,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:33,497 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104402 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:33,497 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:33,497 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,497 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0774fe9157c64e7983a63f492c44cfd3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/21a6415becf243d7a01458396db37e61, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5d507181d5a44813a752f5b48ab4c4b5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/2ac845969a214e8482c44cbde0fa4c61] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=102.0 K 2023-03-19 09:56:33,497 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:33,498 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:33,498 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,498 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/11e8cbdb1cc940d0990690d49114338c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/82ed10b1018443bfae45525c4853ced2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55618be14551471ba1fa301605961b0c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/45e7cb265815401eb7ed03ee244530e3] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=45.4 K 2023-03-19 09:56:33,504 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 11e8cbdb1cc940d0990690d49114338c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679219788595 2023-03-19 09:56:33,504 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0774fe9157c64e7983a63f492c44cfd3, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679219788595 2023-03-19 09:56:33,505 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 82ed10b1018443bfae45525c4853ced2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679219790949 2023-03-19 09:56:33,505 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 21a6415becf243d7a01458396db37e61, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679219790949 2023-03-19 09:56:33,505 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 55618be14551471ba1fa301605961b0c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679219791017 2023-03-19 09:56:33,505 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 5d507181d5a44813a752f5b48ab4c4b5, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679219791017 2023-03-19 09:56:33,505 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 45e7cb265815401eb7ed03ee244530e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679219791442 2023-03-19 09:56:33,505 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 2ac845969a214e8482c44cbde0fa4c61, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679219791442 2023-03-19 09:56:33,522 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#565 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:33,522 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#564 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:33,586 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:33,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:33,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:33,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:33,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:33,590 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:33,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:33,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:33,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,590 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,592 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:33,592 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:33,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:33,593 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:33,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:33,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:33,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:33,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,595 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:33,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:33,595 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-19 09:56:33,596 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:33,596 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:33,596 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:33,596 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:33,596 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:33,596 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:33,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:33,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:33,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219793782, max timestamp=1679219793819Num uniques 859; 2023-03-19 09:56:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219793782, max timestamp=1679219793819Num uniques 859; 2023-03-19 09:56:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219793782, max timestamp=1679219793819Num uniques 859; 2023-03-19 09:56:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:33,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219793873, max timestamp=1679219793894Num uniques 1018; 2023-03-19 09:56:33,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-19 09:56:33,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219793873, max timestamp=1679219793894Num uniques 1018; 2023-03-19 09:56:33,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-19 09:56:33,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679219793873, max timestamp=1679219793894Num uniques 1018; 2023-03-19 09:56:33,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-19 09:56:33,946 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/9ee9dc7a723f4114971474e87385563c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9ee9dc7a723f4114971474e87385563c 2023-03-19 09:56:33,950 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 9ee9dc7a723f4114971474e87385563c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:33,950 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:33,950 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=12, startTime=1679219793496; duration=0sec 2023-03-19 09:56:33,950 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:33,950 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:33,952 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:33,952 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:33,952 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:33,952 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/988b7be44fd2405aa3238e120342b0a1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/daea4e44f5954268950d6c0e2842ecd3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cc007ca695224e189968bd2c9a8147b1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/4f3326e9b69a4dbab70b0f1b92bbe189] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=45.4 K 2023-03-19 09:56:33,952 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 988b7be44fd2405aa3238e120342b0a1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679219788595 2023-03-19 09:56:33,953 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting daea4e44f5954268950d6c0e2842ecd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679219790949 2023-03-19 09:56:33,953 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting cc007ca695224e189968bd2c9a8147b1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679219791017 2023-03-19 09:56:33,953 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 4f3326e9b69a4dbab70b0f1b92bbe189, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679219791442 2023-03-19 09:56:33,969 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#567 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:33,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219793976, max timestamp=1679219793993Num uniques 1027; 2023-03-19 09:56:33,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-19 09:56:33,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219793976, max timestamp=1679219793993Num uniques 1027; 2023-03-19 09:56:33,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-19 09:56:33,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219793976, max timestamp=1679219793993Num uniques 1009; 2023-03-19 09:56:33,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-19 09:56:34,014 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031901645ed2078e4131a2b50ecff6a05406 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031901645ed2078e4131a2b50ecff6a05406 2023-03-19 09:56:34,051 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/835840e36e5f42f49f6ee1df4888a4a5 2023-03-19 09:56:34,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219794040, max timestamp=1679219794079Num uniques 1036; 2023-03-19 09:56:34,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219794040, max timestamp=1679219794079Num uniques 1036; 2023-03-19 09:56:34,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-19 09:56:34,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-19 09:56:34,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219794040, max timestamp=1679219794079Num uniques 1036; 2023-03-19 09:56:34,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-19 09:56:34,102 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/6ed738c8df2f40d5bb0d4810a4a571ef 2023-03-19 09:56:34,145 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/448bc40bc7f049ebbddd6b718239e50e 2023-03-19 09:56:34,151 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/835840e36e5f42f49f6ee1df4888a4a5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/835840e36e5f42f49f6ee1df4888a4a5 2023-03-19 09:56:34,154 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/835840e36e5f42f49f6ee1df4888a4a5, entries=150, sequenceid=321, filesize=25.4 K 2023-03-19 09:56:34,156 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/6ed738c8df2f40d5bb0d4810a4a571ef as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6ed738c8df2f40d5bb0d4810a4a571ef 2023-03-19 09:56:34,160 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6ed738c8df2f40d5bb0d4810a4a571ef, entries=150, sequenceid=321, filesize=11.9 K 2023-03-19 09:56:34,160 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/448bc40bc7f049ebbddd6b718239e50e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/448bc40bc7f049ebbddd6b718239e50e 2023-03-19 09:56:34,164 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/448bc40bc7f049ebbddd6b718239e50e, entries=150, sequenceid=321, filesize=11.9 K 2023-03-19 09:56:34,165 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=127.47 KB/130530 for a391bedbca383dedc4bc8fc94dd92901 in 570ms, sequenceid=321, compaction requested=false 2023-03-19 09:56:34,165 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:34,165 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,165 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:34,165 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:34,165 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,172 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:34,172 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:34,173 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,173 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,174 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=134.18 KB heapSize=382.45 KB 2023-03-19 09:56:34,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:34,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:34,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:34,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:34,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:34,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:34,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:34,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:34,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef1147[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:34,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:34,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,181 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:34,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:34,181 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:34,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,182 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:34,183 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:34,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,183 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:34,189 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:34,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:34,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:34,189 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:34,189 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:34,192 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,192 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,192 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,193 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,194 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,199 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:34,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:34,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,199 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,200 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,200 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:34,200 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:34,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:34,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:34,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219854218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,243 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031946f10a5cc07c4fad9021f3b30a1f56bb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031946f10a5cc07c4fad9021f3b30a1f56bb 2023-03-19 09:56:34,264 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/6c7a3ee77a7145f3827d3d880d73fd30 2023-03-19 09:56:34,300 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:34,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:34,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:34,324 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:34,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:34,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:34,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,325 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:34,330 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:34,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:34,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219854332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:34,372 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:34,372 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,375 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:34,375 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:34,375 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:34,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:34,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,383 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:34,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79fc4053[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:34,385 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:34,385 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,386 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:34,386 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,386 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:34,388 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,391 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/b4882f54a30f425a8d49c84533495d95 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b4882f54a30f425a8d49c84533495d95 2023-03-19 09:56:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:34,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:34,393 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:34,393 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:34,394 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:34,394 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:34,394 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:34,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,395 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,397 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,399 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into b4882f54a30f425a8d49c84533495d95(size=25.9 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:34,399 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:34,399 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=12, startTime=1679219793496; duration=0sec 2023-03-19 09:56:34,400 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,400 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,410 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,411 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,411 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:34,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:34,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,411 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:34,411 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:34,418 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b485aaca876342e881842a142299e526 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b485aaca876342e881842a142299e526 2023-03-19 09:56:34,423 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into b485aaca876342e881842a142299e526(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:34,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:34,423 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=12, startTime=1679219793496; duration=0sec 2023-03-19 09:56:34,423 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:34,511 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:34,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:34,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:34,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:34,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:34,520 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:34,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:34,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:34,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:34,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,524 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:34,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:34,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:34,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:34,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,524 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:34,530 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,531 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@639f1bc5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:34,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:34,533 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,534 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:34,534 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,535 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:34,536 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:34,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:34,539 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:34,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:34,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:34,539 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:34,539 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:34,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:34,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219854536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,541 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,541 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,542 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,543 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,547 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,547 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:34,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:34,547 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:34,547 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:34,648 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:34,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:34,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:34,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:34,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:34,656 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:34,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:34,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:34,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,656 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,657 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:34,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,659 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:34,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:34,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:34,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,672 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:34,673 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:34,673 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:34,675 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:34,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:34,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725e8e50[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:34,682 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:34,682 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,683 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:34,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:34,683 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:34,685 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:34,686 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:34,687 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,687 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,687 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,700 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,700 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,700 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,700 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,700 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,701 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,701 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:34,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:34,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,701 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:34,701 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:34,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e959440e89514f92b4665679b7d98f7e 2023-03-19 09:56:34,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/459d53b87c714a01958b0067177c09f2 2023-03-19 09:56:34,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/6c7a3ee77a7145f3827d3d880d73fd30 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/6c7a3ee77a7145f3827d3d880d73fd30 2023-03-19 09:56:34,794 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/6c7a3ee77a7145f3827d3d880d73fd30, entries=200, sequenceid=410, filesize=32.3 K 2023-03-19 09:56:34,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e959440e89514f92b4665679b7d98f7e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e959440e89514f92b4665679b7d98f7e 2023-03-19 09:56:34,801 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:34,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e959440e89514f92b4665679b7d98f7e, entries=150, sequenceid=410, filesize=11.9 K 2023-03-19 09:56:34,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:34,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/459d53b87c714a01958b0067177c09f2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/459d53b87c714a01958b0067177c09f2 2023-03-19 09:56:34,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:34,805 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:34,805 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,805 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:34,805 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:34,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,806 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/459d53b87c714a01958b0067177c09f2, entries=150, sequenceid=410, filesize=11.9 K 2023-03-19 09:56:34,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~382.41 KB/391584, currentSize=46.96 KB/48090 for a391bedbca383dedc4bc8fc94dd92901 in 631ms, sequenceid=410, compaction requested=true 2023-03-19 09:56:34,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:34,807 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:34,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:34,807 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:34,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:34,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:34,808 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:34,808 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:34,808 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:34,808 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:34,808 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,809 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,809 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b4882f54a30f425a8d49c84533495d95, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/835840e36e5f42f49f6ee1df4888a4a5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/6c7a3ee77a7145f3827d3d880d73fd30] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=83.7 K 2023-03-19 09:56:34,809 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9ee9dc7a723f4114971474e87385563c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6ed738c8df2f40d5bb0d4810a4a571ef, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e959440e89514f92b4665679b7d98f7e] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.2 K 2023-03-19 09:56:34,809 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b4882f54a30f425a8d49c84533495d95, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679219791442 2023-03-19 09:56:34,809 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 835840e36e5f42f49f6ee1df4888a4a5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679219792601 2023-03-19 09:56:34,809 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9ee9dc7a723f4114971474e87385563c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679219791442 2023-03-19 09:56:34,809 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 6c7a3ee77a7145f3827d3d880d73fd30, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679219794111 2023-03-19 09:56:34,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 6ed738c8df2f40d5bb0d4810a4a571ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679219792601 2023-03-19 09:56:34,810 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting e959440e89514f92b4665679b7d98f7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679219794111 2023-03-19 09:56:34,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,815 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,816 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:34,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:34,834 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#573 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:34,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:34,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:34,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:34,840 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:34,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:34,843 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:56:34,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:34,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:34,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:34,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:34,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:34,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:34,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:34,868 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:34,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:34,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:34,868 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:34,869 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:34,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:34,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@940cc75[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:34,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:34,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,882 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,883 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:34,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:34,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:34,883 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:34,885 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:34,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:34,887 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:34,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:34,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:34,887 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:34,887 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:34,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,889 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:34,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:34,891 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:34,894 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:34,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:34,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:34,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:34,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:34,895 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:34,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:34,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:34,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:34,895 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:34,895 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:34,895 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:34,897 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#574 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:34,924 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e0d5b345c39540ea81a8fd82d0a053e3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0d5b345c39540ea81a8fd82d0a053e3 2023-03-19 09:56:34,929 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into e0d5b345c39540ea81a8fd82d0a053e3(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:34,929 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:34,929 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=13, startTime=1679219794807; duration=0sec 2023-03-19 09:56:34,929 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:34,929 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:34,930 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:34,930 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:34,930 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:34,930 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b485aaca876342e881842a142299e526, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/448bc40bc7f049ebbddd6b718239e50e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/459d53b87c714a01958b0067177c09f2] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.2 K 2023-03-19 09:56:34,931 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b485aaca876342e881842a142299e526, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679219791442 2023-03-19 09:56:34,932 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 448bc40bc7f049ebbddd6b718239e50e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679219792601 2023-03-19 09:56:34,932 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 459d53b87c714a01958b0067177c09f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679219794111 2023-03-19 09:56:34,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219794909, max timestamp=1679219794956Num uniques 859; 2023-03-19 09:56:34,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:34,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219794909, max timestamp=1679219794956Num uniques 859; 2023-03-19 09:56:34,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:34,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219794909, max timestamp=1679219794956Num uniques 859; 2023-03-19 09:56:34,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:34,985 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#576 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:34,991 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303196c1218359c6f4e03b5210a15bb71eade to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196c1218359c6f4e03b5210a15bb71eade 2023-03-19 09:56:34,995 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:34,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:34,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:35,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:35,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:35,012 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:35,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:35,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:35,012 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:35,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,013 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,015 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:35,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:35,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,016 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:35,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:35,037 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/5ad658edaa10461aa9b260c04c76ca05 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/5ad658edaa10461aa9b260c04c76ca05 2023-03-19 09:56:35,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:35,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:35,042 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 5ad658edaa10461aa9b260c04c76ca05(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:35,042 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:35,042 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=13, startTime=1679219794807; duration=0sec 2023-03-19 09:56:35,042 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:35,049 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:35,049 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:35,049 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:35,049 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:35,050 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=423, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/9fe2909631e74e39b92351dd6caab938 2023-03-19 09:56:35,051 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:35,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:35,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1225ebd6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:35,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:35,053 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,055 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:35,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:35,055 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:35,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219855045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,055 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:35,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:35,057 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:35,057 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:35,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:35,059 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:35,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:35,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:35,059 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:35,059 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:35,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,061 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:35,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:35,062 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,063 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:35,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,068 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:35,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:35,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:35,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:35,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:35,069 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:35,069 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:35,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/ceafb150f8d84b6bbef00bb649970e81 2023-03-19 09:56:35,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/0041e98ddae341bab7e8480d961ce41b 2023-03-19 09:56:35,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219855156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/9fe2909631e74e39b92351dd6caab938 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9fe2909631e74e39b92351dd6caab938 2023-03-19 09:56:35,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9fe2909631e74e39b92351dd6caab938, entries=200, sequenceid=423, filesize=32.3 K 2023-03-19 09:56:35,169 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:35,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/ceafb150f8d84b6bbef00bb649970e81 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/ceafb150f8d84b6bbef00bb649970e81 2023-03-19 09:56:35,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:35,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/ceafb150f8d84b6bbef00bb649970e81, entries=150, sequenceid=423, filesize=11.9 K 2023-03-19 09:56:35,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:35,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/0041e98ddae341bab7e8480d961ce41b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0041e98ddae341bab7e8480d961ce41b 2023-03-19 09:56:35,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:35,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:35,176 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:35,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:35,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:35,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:35,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,178 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:35,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:35,179 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,179 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,179 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:35,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0041e98ddae341bab7e8480d961ce41b, entries=150, sequenceid=423, filesize=11.9 K 2023-03-19 09:56:35,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:35,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:35,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a391bedbca383dedc4bc8fc94dd92901 in 347ms, sequenceid=423, compaction requested=false 2023-03-19 09:56:35,190 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:35,191 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:35,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:35,191 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:35,191 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:35,191 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:35,191 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:35,191 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:35,191 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-19 09:56:35,191 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:35,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:35,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:35,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:35,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:35,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:35,243 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319a8bd13bbab0c4eb897d76810c0056876 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a8bd13bbab0c4eb897d76810c0056876 2023-03-19 09:56:35,280 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/849521f4f5ee4205870e4d9b2bc2f896 2023-03-19 09:56:35,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:35,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:35,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219855409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219855513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36760 deadline: 1679219855667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36720 deadline: 1679219855667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,673 DEBUG [Thread-3957] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:35,678 DEBUG [Thread-3953] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:35,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36776 deadline: 1679219855691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,701 DEBUG [Thread-3959] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:35,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36710 deadline: 1679219855709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,716 DEBUG [Thread-3951] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8198 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., hostname=jenkins-hbase4.apache.org,34669,1679219618625, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-19 09:56:35,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:35,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219855716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,753 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/03005e846c7f4ef294e418388bbbdade 2023-03-19 09:56:35,786 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b039183288404041967e37e17447b92c 2023-03-19 09:56:35,791 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/849521f4f5ee4205870e4d9b2bc2f896 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/849521f4f5ee4205870e4d9b2bc2f896 2023-03-19 09:56:35,795 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/849521f4f5ee4205870e4d9b2bc2f896, entries=150, sequenceid=461, filesize=25.4 K 2023-03-19 09:56:35,796 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/03005e846c7f4ef294e418388bbbdade as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/03005e846c7f4ef294e418388bbbdade 2023-03-19 09:56:35,802 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/03005e846c7f4ef294e418388bbbdade, entries=150, sequenceid=461, filesize=11.9 K 2023-03-19 09:56:35,802 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b039183288404041967e37e17447b92c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b039183288404041967e37e17447b92c 2023-03-19 09:56:35,806 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/bc33dfba55f44f85819d54cc7fa57424 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/bc33dfba55f44f85819d54cc7fa57424 2023-03-19 09:56:35,810 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b039183288404041967e37e17447b92c, entries=150, sequenceid=461, filesize=11.9 K 2023-03-19 09:56:35,810 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into bc33dfba55f44f85819d54cc7fa57424(size=26.0 K), total size for store is 83.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-19 09:56:35,811 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:35,811 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=13, startTime=1679219794807; duration=1sec 2023-03-19 09:56:35,811 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:35,811 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a391bedbca383dedc4bc8fc94dd92901 in 620ms, sequenceid=461, compaction requested=true 2023-03-19 09:56:35,811 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:35,811 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:35,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,813 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:35,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,813 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97f8b64[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:35,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:35,815 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:35,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,818 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:35,819 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:35,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:35,822 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:35,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:35,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:35,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:35,823 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:35,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,824 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:35,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:35,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:35,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:35,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:35,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:35,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:35,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,829 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:35,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:35,828 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:35,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:35,829 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:35,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:35,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,929 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:35,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:35,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:35,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:35,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:35,936 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:35,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:35,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:35,936 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:35,937 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,937 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,939 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:35,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:35,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:35,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:35,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:35,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:35,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:35,944 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:35,944 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:35,944 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:35,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:35,947 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:35,947 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:35,948 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:35,958 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:56:35,958 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:35,958 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:35,959 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:35,959 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:35,959 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:35,959 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:35,985 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319b7888e5698774b0fbe92a413788be339 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b7888e5698774b0fbe92a413788be339 2023-03-19 09:56:35,990 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=473, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/0819bad8e6b9447e8a3abab79f5695ca 2023-03-19 09:56:36,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:36,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:36,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219796075, max timestamp=1679219796086Num uniques 859; 2023-03-19 09:56:36,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219796075, max timestamp=1679219796086Num uniques 859; 2023-03-19 09:56:36,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219796075, max timestamp=1679219796086Num uniques 859; 2023-03-19 09:56:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:36,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:36,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:36,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219856165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:36,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219856271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,406 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/d02ab416674046769b5cb5cbe9362ece 2023-03-19 09:56:36,416 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b499e1a8922a457296899f6325a70d48 2023-03-19 09:56:36,420 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/0819bad8e6b9447e8a3abab79f5695ca as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0819bad8e6b9447e8a3abab79f5695ca 2023-03-19 09:56:36,423 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0819bad8e6b9447e8a3abab79f5695ca, entries=150, sequenceid=473, filesize=25.4 K 2023-03-19 09:56:36,424 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/d02ab416674046769b5cb5cbe9362ece as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d02ab416674046769b5cb5cbe9362ece 2023-03-19 09:56:36,427 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d02ab416674046769b5cb5cbe9362ece, entries=150, sequenceid=473, filesize=11.9 K 2023-03-19 09:56:36,427 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b499e1a8922a457296899f6325a70d48 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b499e1a8922a457296899f6325a70d48 2023-03-19 09:56:36,431 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b499e1a8922a457296899f6325a70d48, entries=150, sequenceid=473, filesize=11.9 K 2023-03-19 09:56:36,431 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a391bedbca383dedc4bc8fc94dd92901 in 483ms, sequenceid=473, compaction requested=true 2023-03-19 09:56:36,431 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:36,431 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:36,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:36,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:36,432 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:36,436 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,436 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,437 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:36,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd5b815[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:36,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:36,438 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,440 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,440 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:36,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,441 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:36,441 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:36,441 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:36,443 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,443 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:36,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:36,446 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:36,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:36,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:36,446 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:36,446 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:36,447 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,447 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,447 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,448 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,449 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:36,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:36,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:36,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:36,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:36,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,452 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:36,452 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:36,451 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,451 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:36,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,452 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:36,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:56:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:36,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:36,507 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319ce645a2876ba45e395f0488faf0cf942 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ce645a2876ba45e395f0488faf0cf942 2023-03-19 09:56:36,514 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/7b4b41ef1a964af388c7dfab72d1fe85 2023-03-19 09:56:36,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:36,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219856519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/9bd021d3c44647c4bbbe289112ffff8e 2023-03-19 09:56:36,552 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:36,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:36,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:36,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/09b5db4d7a4d411dbf1328c550a408aa 2023-03-19 09:56:36,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:36,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:36,556 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:36,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:36,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:36,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:36,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,556 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,558 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:36,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:36,558 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,558 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,558 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:36,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:36,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/7b4b41ef1a964af388c7dfab72d1fe85 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7b4b41ef1a964af388c7dfab72d1fe85 2023-03-19 09:56:36,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:36,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:36,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:36,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:36,566 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:36,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7b4b41ef1a964af388c7dfab72d1fe85, entries=200, sequenceid=510, filesize=32.3 K 2023-03-19 09:56:36,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/9bd021d3c44647c4bbbe289112ffff8e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9bd021d3c44647c4bbbe289112ffff8e 2023-03-19 09:56:36,576 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:36,576 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,576 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:36,576 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:36,576 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:36,576 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,576 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:36,576 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:36,576 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,586 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:36,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9bd021d3c44647c4bbbe289112ffff8e, entries=150, sequenceid=510, filesize=11.9 K 2023-03-19 09:56:36,591 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,591 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/09b5db4d7a4d411dbf1328c550a408aa as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09b5db4d7a4d411dbf1328c550a408aa 2023-03-19 09:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,592 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,592 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:36,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f9fefe[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:36,594 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:36,594 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,595 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,595 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:36,595 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:36,596 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:36,596 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:36,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09b5db4d7a4d411dbf1328c550a408aa, entries=150, sequenceid=510, filesize=11.9 K 2023-03-19 09:56:36,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for a391bedbca383dedc4bc8fc94dd92901 in 111ms, sequenceid=510, compaction requested=true 2023-03-19 09:56:36,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:36,597 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:36,597 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:36,597 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:56:36,597 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:56:36,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:36,597 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,597 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:36,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:36,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:36,600 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144924 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:56:36,600 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:36,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:36,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:36,600 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:36,600 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:36,600 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:36,601 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,601 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/bc33dfba55f44f85819d54cc7fa57424, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9fe2909631e74e39b92351dd6caab938, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/849521f4f5ee4205870e4d9b2bc2f896, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0819bad8e6b9447e8a3abab79f5695ca, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7b4b41ef1a964af388c7dfab72d1fe85] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=141.5 K 2023-03-19 09:56:36,602 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:56:36,602 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:36,602 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,602 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0d5b345c39540ea81a8fd82d0a053e3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/ceafb150f8d84b6bbef00bb649970e81, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/03005e846c7f4ef294e418388bbbdade, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d02ab416674046769b5cb5cbe9362ece, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9bd021d3c44647c4bbbe289112ffff8e] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=60.1 K 2023-03-19 09:56:36,602 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting bc33dfba55f44f85819d54cc7fa57424, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679219794111 2023-03-19 09:56:36,602 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,602 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,602 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,602 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,606 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,609 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e0d5b345c39540ea81a8fd82d0a053e3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679219794111 2023-03-19 09:56:36,609 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9fe2909631e74e39b92351dd6caab938, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1679219794197 2023-03-19 09:56:36,610 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:36,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:36,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:36,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,610 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:36,610 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:36,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:36,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:36,610 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,611 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,611 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,611 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,611 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:36,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,612 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting ceafb150f8d84b6bbef00bb649970e81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1679219794210 2023-03-19 09:56:36,612 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 849521f4f5ee4205870e4d9b2bc2f896, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679219795014 2023-03-19 09:56:36,612 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 03005e846c7f4ef294e418388bbbdade, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679219795014 2023-03-19 09:56:36,612 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 0819bad8e6b9447e8a3abab79f5695ca, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1679219795392 2023-03-19 09:56:36,613 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d02ab416674046769b5cb5cbe9362ece, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1679219795392 2023-03-19 09:56:36,615 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 7b4b41ef1a964af388c7dfab72d1fe85, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679219796135 2023-03-19 09:56:36,615 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9bd021d3c44647c4bbbe289112ffff8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679219796135 2023-03-19 09:56:36,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:56:36,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:36,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:36,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:36,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:36,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:36,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:36,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:36,630 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#589 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:36,666 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#588 average throughput is 0.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:36,683 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/f63364bed50b46bb80eb61d69417015b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/f63364bed50b46bb80eb61d69417015b 2023-03-19 09:56:36,688 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into f63364bed50b46bb80eb61d69417015b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:36,688 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:36,688 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=11, startTime=1679219796597; duration=0sec 2023-03-19 09:56:36,688 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:36,688 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:56:36,690 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:56:36,690 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:36,690 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,690 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/5ad658edaa10461aa9b260c04c76ca05, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0041e98ddae341bab7e8480d961ce41b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b039183288404041967e37e17447b92c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b499e1a8922a457296899f6325a70d48, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09b5db4d7a4d411dbf1328c550a408aa] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=60.1 K 2023-03-19 09:56:36,690 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 5ad658edaa10461aa9b260c04c76ca05, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679219794111 2023-03-19 09:56:36,691 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0041e98ddae341bab7e8480d961ce41b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1679219794210 2023-03-19 09:56:36,691 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b039183288404041967e37e17447b92c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679219795014 2023-03-19 09:56:36,691 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b499e1a8922a457296899f6325a70d48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1679219795392 2023-03-19 09:56:36,692 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 09b5db4d7a4d411dbf1328c550a408aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679219796135 2023-03-19 09:56:36,698 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#591 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:36,704 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319a481b4c3eaec4ce49df35b55d60e2d7f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a481b4c3eaec4ce49df35b55d60e2d7f 2023-03-19 09:56:36,710 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:36,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:36,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:36,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:36,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:36,724 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:36,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:36,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:36,724 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:36,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,725 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,726 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:36,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219796676, max timestamp=1679219796728Num uniques 859; 2023-03-19 09:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:36,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219796676, max timestamp=1679219796728Num uniques 859; 2023-03-19 09:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:36,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219796676, max timestamp=1679219796728Num uniques 859; 2023-03-19 09:56:36,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:36,729 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,736 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,737 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:36,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:36,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:36,751 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,751 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:36,751 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=521, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/211e88428635494ab38026a9322b9fac 2023-03-19 09:56:36,751 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:36,751 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:36,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,754 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:36,759 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:36,759 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3fde6c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:36,760 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,761 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:36,761 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:36,761 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:36,761 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,762 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:36,765 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:36,769 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:36,769 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:36,769 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:36,769 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:36,769 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:36,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:36,770 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,770 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,771 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,773 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,777 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:36,778 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:36,778 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:36,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:36,779 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:36,779 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:36,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/36a14cb330f04b739b8343eb25613da9 2023-03-19 09:56:36,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/af3c9dc380f44090acb20482ca6205c2 2023-03-19 09:56:36,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/211e88428635494ab38026a9322b9fac as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/211e88428635494ab38026a9322b9fac 2023-03-19 09:56:36,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/211e88428635494ab38026a9322b9fac, entries=200, sequenceid=521, filesize=32.3 K 2023-03-19 09:56:36,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/36a14cb330f04b739b8343eb25613da9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/36a14cb330f04b739b8343eb25613da9 2023-03-19 09:56:36,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/36a14cb330f04b739b8343eb25613da9, entries=150, sequenceid=521, filesize=11.9 K 2023-03-19 09:56:36,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/af3c9dc380f44090acb20482ca6205c2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/af3c9dc380f44090acb20482ca6205c2 2023-03-19 09:56:36,823 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/af3c9dc380f44090acb20482ca6205c2, entries=150, sequenceid=521, filesize=11.9 K 2023-03-19 09:56:36,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=114.05 KB/116790 for a391bedbca383dedc4bc8fc94dd92901 in 195ms, sequenceid=521, compaction requested=false 2023-03-19 09:56:36,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:36,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:36,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-03-19 09:56:36,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:36,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:36,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:36,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:36,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:36,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:36,868 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023031993d370f794104befaab554b4602edd52 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031993d370f794104befaab554b4602edd52 2023-03-19 09:56:36,879 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:36,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:36,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:36,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:36,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219856886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:36,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:36,896 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:36,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:36,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:36,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:36,896 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,897 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,898 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:36,898 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,898 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,898 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:36,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:36,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:36,912 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:36,912 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:36,912 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:36,920 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:36,920 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:36,920 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=557, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/01fb137e97e44246ab5e8063398f3b13 2023-03-19 09:56:36,921 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,921 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:36,922 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:36,922 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:36,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:36,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:36,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:36,922 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:36,925 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,925 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,925 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,925 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:36,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486e4604[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:36,927 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:36,928 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:36,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:36,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:36,933 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:36,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:36,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:36,933 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:36,933 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:36,936 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,936 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,937 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:36,937 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,940 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:36,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:36,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:36,943 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:36,942 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:36,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:36,943 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:36,943 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:36,943 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:36,942 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:36,943 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,943 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:36,943 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:36,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:36,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/b4cde5dc84174eb3992b7dd77c6488d9 2023-03-19 09:56:36,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/9b28c17a00ea44cf8c0c5177bc8d5c8f 2023-03-19 09:56:36,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/01fb137e97e44246ab5e8063398f3b13 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/01fb137e97e44246ab5e8063398f3b13 2023-03-19 09:56:36,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/01fb137e97e44246ab5e8063398f3b13, entries=200, sequenceid=557, filesize=32.3 K 2023-03-19 09:56:36,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/b4cde5dc84174eb3992b7dd77c6488d9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b4cde5dc84174eb3992b7dd77c6488d9 2023-03-19 09:56:37,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b4cde5dc84174eb3992b7dd77c6488d9, entries=150, sequenceid=557, filesize=11.9 K 2023-03-19 09:56:37,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:37,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/9b28c17a00ea44cf8c0c5177bc8d5c8f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9b28c17a00ea44cf8c0c5177bc8d5c8f 2023-03-19 09:56:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219856997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9b28c17a00ea44cf8c0c5177bc8d5c8f, entries=150, sequenceid=557, filesize=11.9 K 2023-03-19 09:56:37,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for a391bedbca383dedc4bc8fc94dd92901 in 190ms, sequenceid=557, compaction requested=true 2023-03-19 09:56:37,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:37,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:37,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-19 09:56:37,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-19 09:56:37,043 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:37,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:37,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:37,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:37,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:37,047 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:37,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:37,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:37,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,048 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,049 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,049 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,050 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:37,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:37,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:37,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:37,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:37,068 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:37,068 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,068 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:37,068 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-19 09:56:37,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:37,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:37,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:37,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:37,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:37,070 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:37,126 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/1d674337aa324eacb4f52765a216094d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1d674337aa324eacb4f52765a216094d 2023-03-19 09:56:37,127 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319cd61854b80ec4da9b070dc368f535845 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319cd61854b80ec4da9b070dc368f535845 2023-03-19 09:56:37,135 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 1d674337aa324eacb4f52765a216094d(size=12.7 K), total size for store is 36.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:37,135 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:37,135 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=11, startTime=1679219796597; duration=0sec 2023-03-19 09:56:37,135 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-19 09:56:37,135 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 5 compacting, 2 eligible, 16 blocking 2023-03-19 09:56:37,140 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-19 09:56:37,140 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-19 09:56:37,140 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. because compaction request was cancelled 2023-03-19 09:56:37,140 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:37,142 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:37,142 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:37,142 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,142 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/f63364bed50b46bb80eb61d69417015b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/36a14cb330f04b739b8343eb25613da9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b4cde5dc84174eb3992b7dd77c6488d9] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.5 K 2023-03-19 09:56:37,143 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting f63364bed50b46bb80eb61d69417015b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679219796135 2023-03-19 09:56:37,144 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 36a14cb330f04b739b8343eb25613da9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1679219796512 2023-03-19 09:56:37,144 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b4cde5dc84174eb3992b7dd77c6488d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1679219796778 2023-03-19 09:56:37,149 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=569, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/92438a70ca2b41d2a075749d70791d19 2023-03-19 09:56:37,151 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/930a23777965496d87c0ec251809f91b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/930a23777965496d87c0ec251809f91b 2023-03-19 09:56:37,157 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#598 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:37,191 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into 930a23777965496d87c0ec251809f91b(size=26.2 K), total size for store is 90.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:37,191 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:37,191 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=11, startTime=1679219796597; duration=0sec 2023-03-19 09:56:37,191 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:37,191 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:37,192 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:37,192 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:37,193 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,193 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1d674337aa324eacb4f52765a216094d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/af3c9dc380f44090acb20482ca6205c2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9b28c17a00ea44cf8c0c5177bc8d5c8f] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.5 K 2023-03-19 09:56:37,193 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1d674337aa324eacb4f52765a216094d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679219796135 2023-03-19 09:56:37,194 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting af3c9dc380f44090acb20482ca6205c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1679219796512 2023-03-19 09:56:37,194 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9b28c17a00ea44cf8c0c5177bc8d5c8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1679219796778 2023-03-19 09:56:37,199 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/21ab2ab0e688407b8159be2c612f7fc3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/21ab2ab0e688407b8159be2c612f7fc3 2023-03-19 09:56:37,205 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#600 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:37,205 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 21ab2ab0e688407b8159be2c612f7fc3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:37,205 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:37,205 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=13, startTime=1679219797015; duration=0sec 2023-03-19 09:56:37,205 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:37,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:37,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:37,209 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5 2023-03-19 09:56:37,218 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/33bb3ac00a504444bd6eda6b30d005f5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/33bb3ac00a504444bd6eda6b30d005f5 2023-03-19 09:56:37,221 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 33bb3ac00a504444bd6eda6b30d005f5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:37,222 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:37,222 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=13, startTime=1679219797015; duration=0sec 2023-03-19 09:56:37,222 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:37,228 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/a4dd03d584a847538ab55f27379d0c1e 2023-03-19 09:56:37,232 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/92438a70ca2b41d2a075749d70791d19 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/92438a70ca2b41d2a075749d70791d19 2023-03-19 09:56:37,235 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/92438a70ca2b41d2a075749d70791d19, entries=150, sequenceid=569, filesize=25.4 K 2023-03-19 09:56:37,238 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5 2023-03-19 09:56:37,241 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5, entries=150, sequenceid=569, filesize=11.9 K 2023-03-19 09:56:37,242 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/a4dd03d584a847538ab55f27379d0c1e as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a4dd03d584a847538ab55f27379d0c1e 2023-03-19 09:56:37,249 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a4dd03d584a847538ab55f27379d0c1e, entries=150, sequenceid=569, filesize=11.9 K 2023-03-19 09:56:37,250 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=46.96 KB/48090 for a391bedbca383dedc4bc8fc94dd92901 in 182ms, sequenceid=569, compaction requested=true 2023-03-19 09:56:37,250 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:37,250 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:37,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:37,250 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:37,255 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:37,256 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,256 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,257 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,258 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-19 09:56:37,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:37,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:37,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:37,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:37,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:37,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:37,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:37,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad95a0c[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:37,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:37,260 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,263 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:37,263 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:37,263 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:37,265 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:37,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:37,267 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:37,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:37,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:37,267 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:37,267 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:37,269 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,269 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,269 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,271 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,274 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,276 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,276 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,276 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,277 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:37,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:37,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,277 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,280 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:37,280 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:37,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219797298, max timestamp=1679219797338Num uniques 859; 2023-03-19 09:56:37,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:37,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219797298, max timestamp=1679219797338Num uniques 859; 2023-03-19 09:56:37,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:37,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219797298, max timestamp=1679219797338Num uniques 859; 2023-03-19 09:56:37,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:37,380 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:37,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:37,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:37,384 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:37,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:37,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:37,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,387 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,387 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:37,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:37,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:37,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:37,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:37,392 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:37,392 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,393 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:37,404 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:37,404 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:37,406 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,407 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,407 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251857a7[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:37,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:37,414 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,415 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:37,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:37,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,416 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:37,416 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:37,417 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:37,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,417 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:37,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219857419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:37,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:37,430 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:37,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:37,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:37,430 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:37,430 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:37,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,433 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,434 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,435 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,437 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,437 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,437 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,437 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:37,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:37,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,437 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,437 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,438 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,438 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:37,438 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:37,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:37,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219857525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,538 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:37,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:37,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:37,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:37,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:37,541 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:37,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:37,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:37,541 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,542 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,543 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,543 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,543 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,543 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:37,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:37,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:37,545 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,545 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:37,545 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:37,545 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:37,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:37,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:37,545 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:37,546 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,546 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,547 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:37,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@669731d6[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:37,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:37,548 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,550 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,551 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:37,551 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:37,551 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:37,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,551 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:37,552 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:37,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,552 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:37,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:37,555 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:37,555 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:37,555 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:37,555 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:37,555 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:37,559 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,559 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,561 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,562 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,563 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,567 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:37,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:37,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,567 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:37,567 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:37,567 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:37,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,667 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:37,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:37,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:37,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:37,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:37,671 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:37,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:37,671 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:37,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,672 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,680 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,680 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:37,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:37,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:37,682 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:37,682 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,682 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:37,682 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:37,685 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,687 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:37,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c731566[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:37,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:37,689 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,690 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:37,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,691 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:37,693 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:37,693 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,693 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,693 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:37,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:37,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:37,696 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:37,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:37,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:37,696 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:37,696 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:37,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,699 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,700 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,702 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,705 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:37,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:37,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,706 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,706 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,706 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,706 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:37,706 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,705 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,708 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303193818d7e0683d46d8b6a351acaac27ce2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193818d7e0683d46d8b6a351acaac27ce2 2023-03-19 09:56:37,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:37,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219857728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,806 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:37,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:37,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:37,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:37,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:37,810 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:37,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:37,810 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:37,811 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,811 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,812 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,813 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:37,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:37,814 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,815 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:37,815 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:37,815 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,815 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:37,815 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:37,815 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,815 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:37,815 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:37,817 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,817 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cda2dde[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:37,819 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:37,820 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,821 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,821 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:37,821 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:37,821 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:37,821 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,822 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:37,822 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,822 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:37,822 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,822 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:37,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:37,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:37,825 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:37,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:37,825 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:37,826 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:37,826 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:37,827 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,828 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,829 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,831 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,834 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,835 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:37,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:37,836 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:37,836 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:37,936 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:37,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:37,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:37,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:37,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:37,939 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:37,939 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,939 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:37,939 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:37,939 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,939 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,941 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,941 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,941 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,941 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:37,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:37,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:37,942 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:37,942 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:37,942 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:37,942 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:37,942 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:37,942 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,942 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:37,943 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:37,943 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:37,944 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b18f88d[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:37,946 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:37,946 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,948 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:37,948 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,948 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:37,950 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,950 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:37,950 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,950 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:37,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:37,952 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:37,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:37,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:37,952 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:37,952 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:37,954 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,954 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,954 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:37,954 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:37,956 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:37,960 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:37,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:37,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:37,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:37,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:37,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:37,961 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:37,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:37,960 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:37,961 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:37,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:37,961 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:37,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:38,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 483 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219858032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,061 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:38,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:38,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:38,068 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:38,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:38,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:38,068 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,069 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,070 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,070 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,071 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:38,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:38,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:38,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:38,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:38,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:38,084 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:38,086 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,086 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,086 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,086 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:38,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:38,089 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445e73b9[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:38,090 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:38,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:38,091 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:38,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,091 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:38,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:38,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,092 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:38,093 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:38,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:38,098 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:38,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:38,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:38,098 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:38,098 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:38,100 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,101 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,102 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,104 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,108 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,109 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,109 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:38,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:38,110 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:38,110 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:38,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:38,139 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=584, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/e75bf80099d5417a8858e126b17685e5 2023-03-19 09:56:38,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e8798ce71fcc40e88159e6716fd9d0d9 2023-03-19 09:56:38,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/8fa0725e2a494f078c6a728300db3789 2023-03-19 09:56:38,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/e75bf80099d5417a8858e126b17685e5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/e75bf80099d5417a8858e126b17685e5 2023-03-19 09:56:38,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/e75bf80099d5417a8858e126b17685e5, entries=250, sequenceid=584, filesize=39.2 K 2023-03-19 09:56:38,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e8798ce71fcc40e88159e6716fd9d0d9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e8798ce71fcc40e88159e6716fd9d0d9 2023-03-19 09:56:38,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e8798ce71fcc40e88159e6716fd9d0d9, entries=150, sequenceid=584, filesize=11.9 K 2023-03-19 09:56:38,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/8fa0725e2a494f078c6a728300db3789 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/8fa0725e2a494f078c6a728300db3789 2023-03-19 09:56:38,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/8fa0725e2a494f078c6a728300db3789, entries=150, sequenceid=584, filesize=11.9 K 2023-03-19 09:56:38,199 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a391bedbca383dedc4bc8fc94dd92901 in 941ms, sequenceid=584, compaction requested=true 2023-03-19 09:56:38,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:38,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:38,199 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-19 09:56:38,199 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:38,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:38,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:38,200 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:38,201 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:38,201 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,201 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/21ab2ab0e688407b8159be2c612f7fc3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e8798ce71fcc40e88159e6716fd9d0d9] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.6 K 2023-03-19 09:56:38,201 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 21ab2ab0e688407b8159be2c612f7fc3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1679219796778 2023-03-19 09:56:38,201 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159178 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-19 09:56:38,201 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting fc8ec5e7f4d84e47ab0a5ad96fc7bda5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679219796865 2023-03-19 09:56:38,202 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:38,202 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,202 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/930a23777965496d87c0ec251809f91b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/211e88428635494ab38026a9322b9fac, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/01fb137e97e44246ab5e8063398f3b13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/92438a70ca2b41d2a075749d70791d19, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/e75bf80099d5417a8858e126b17685e5] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=155.4 K 2023-03-19 09:56:38,202 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting e8798ce71fcc40e88159e6716fd9d0d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679219797231 2023-03-19 09:56:38,202 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 930a23777965496d87c0ec251809f91b, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1679219796135 2023-03-19 09:56:38,203 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 211e88428635494ab38026a9322b9fac, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1679219796502 2023-03-19 09:56:38,204 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 01fb137e97e44246ab5e8063398f3b13, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1679219796778 2023-03-19 09:56:38,205 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 92438a70ca2b41d2a075749d70791d19, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679219796865 2023-03-19 09:56:38,205 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting e75bf80099d5417a8858e126b17685e5, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679219797212 2023-03-19 09:56:38,209 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#605 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:38,210 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:38,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:38,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:38,214 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:38,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:38,214 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:38,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,215 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,216 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,216 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,217 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:38,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:38,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:38,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:38,232 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,232 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:38,232 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-19 09:56:38,232 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:38,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:38,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:38,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:38,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:38,233 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:38,243 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#606 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:38,252 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/9c1bb2cc6e8d4a27afdf5f2ec207274b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9c1bb2cc6e8d4a27afdf5f2ec207274b 2023-03-19 09:56:38,258 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 9c1bb2cc6e8d4a27afdf5f2ec207274b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:38,258 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:38,258 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=13, startTime=1679219798199; duration=0sec 2023-03-19 09:56:38,258 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:38,258 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:38,264 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:38,264 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:38,264 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,265 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/33bb3ac00a504444bd6eda6b30d005f5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a4dd03d584a847538ab55f27379d0c1e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/8fa0725e2a494f078c6a728300db3789] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.6 K 2023-03-19 09:56:38,265 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 33bb3ac00a504444bd6eda6b30d005f5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1679219796778 2023-03-19 09:56:38,265 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a4dd03d584a847538ab55f27379d0c1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679219796865 2023-03-19 09:56:38,265 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 8fa0725e2a494f078c6a728300db3789, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679219797231 2023-03-19 09:56:38,270 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#608 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:38,306 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319ed3deccbb4c94d0dbf1cd1fcdd8d8b30 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ed3deccbb4c94d0dbf1cd1fcdd8d8b30 2023-03-19 09:56:38,309 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/1bdf55b990eb405f9c103655b8886cc5 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1bdf55b990eb405f9c103655b8886cc5 2023-03-19 09:56:38,314 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 1bdf55b990eb405f9c103655b8886cc5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:38,314 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:38,314 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=13, startTime=1679219798199; duration=0sec 2023-03-19 09:56:38,314 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:38,314 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=620, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/b8a445c932c54ca2a57af08007b0901d 2023-03-19 09:56:38,334 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/d7105a550aec44a0b7b88135c2c292fb 2023-03-19 09:56:38,357 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/13a0cd3db77f4b5fbb717f2f00711e4b 2023-03-19 09:56:38,361 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/b8a445c932c54ca2a57af08007b0901d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b8a445c932c54ca2a57af08007b0901d 2023-03-19 09:56:38,364 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b8a445c932c54ca2a57af08007b0901d, entries=150, sequenceid=620, filesize=25.4 K 2023-03-19 09:56:38,365 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/d7105a550aec44a0b7b88135c2c292fb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7105a550aec44a0b7b88135c2c292fb 2023-03-19 09:56:38,369 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7105a550aec44a0b7b88135c2c292fb, entries=150, sequenceid=620, filesize=11.9 K 2023-03-19 09:56:38,369 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/13a0cd3db77f4b5fbb717f2f00711e4b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/13a0cd3db77f4b5fbb717f2f00711e4b 2023-03-19 09:56:38,377 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/13a0cd3db77f4b5fbb717f2f00711e4b, entries=150, sequenceid=620, filesize=11.9 K 2023-03-19 09:56:38,378 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for a391bedbca383dedc4bc8fc94dd92901 in 146ms, sequenceid=620, compaction requested=false 2023-03-19 09:56:38,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:38,378 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:38,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:38,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:38,378 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:38,380 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,380 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:38,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429671[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:38,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:38,382 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,383 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:38,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:38,384 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:38,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,384 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:38,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:38,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,385 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:38,385 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:38,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:38,388 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:38,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:38,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:38,388 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:38,388 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:38,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,391 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,392 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,394 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,396 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:38,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:38,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:38,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,396 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,397 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:38,397 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:38,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,397 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,497 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:38,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:38,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:38,500 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:38,500 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:38,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:38,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,501 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,502 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,503 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:38,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:38,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:38,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:38,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:38,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:38,504 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:38,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:38,504 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:38,505 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:38,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:38,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:38,505 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:38,506 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,506 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,507 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:38,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ddbff7a[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:38,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:38,509 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,515 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:38,516 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,516 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:38,517 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,517 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,517 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,517 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:38,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:38,520 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:38,520 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:38,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:38,521 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:38,521 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:38,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,522 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,523 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,523 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,525 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,527 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:38,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:38,528 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,528 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:38,528 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:38,528 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:38,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:38,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-19 09:56:38,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:38,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:38,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:38,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:38,628 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:38,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:38,629 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:38,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:38,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:38,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:38,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:38,633 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:38,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:38,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:38,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,633 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:38,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:38,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:38,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:38,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:38,642 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:38,642 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,642 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:38,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:38,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:38,643 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,643 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:38,643 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:38,643 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:38,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:38,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,643 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:38,645 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,645 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,648 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303193b76974f52cd401883429b1f0ec36c84 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193b76974f52cd401883429b1f0ec36c84 2023-03-19 09:56:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:38,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715bb6dc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:38,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:38,649 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,651 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:38,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,652 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:38,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:38,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,654 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:38,654 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:38,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:38,658 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:38,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:38,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:38,658 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:38,658 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:38,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,661 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,663 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,665 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,668 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,668 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:38,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:38,669 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,669 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,669 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,669 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,669 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,669 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:38,669 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:38,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,699 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/494c4c14fd8547d18fcac6d872ab23c2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/494c4c14fd8547d18fcac6d872ab23c2 2023-03-19 09:56:38,704 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into 494c4c14fd8547d18fcac6d872ab23c2(size=26.4 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:38,704 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:38,704 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=11, startTime=1679219798199; duration=0sec 2023-03-19 09:56:38,704 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:38,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219798698, max timestamp=1679219798715Num uniques 859; 2023-03-19 09:56:38,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219798698, max timestamp=1679219798715Num uniques 859; 2023-03-19 09:56:38,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:38,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219798698, max timestamp=1679219798715Num uniques 859; 2023-03-19 09:56:38,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:38,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:38,769 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:38,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:38,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:38,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:38,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 526 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219858769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:38,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:38,773 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:38,773 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,773 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:38,773 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:38,773 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,774 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,775 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,775 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:38,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:38,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:38,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:38,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:38,776 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:38,777 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,777 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:38,777 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:38,777 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:38,780 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,780 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,780 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,780 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:38,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46afc0f5[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:38,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:38,783 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,784 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,784 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:38,784 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:38,784 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:38,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,785 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,786 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:38,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:38,789 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:38,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:38,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:38,789 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:38,789 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:38,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,791 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,792 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,793 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,796 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:38,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:38,796 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,796 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,797 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:38,797 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:38,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,797 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:38,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:38,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 528 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219858874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,897 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:38,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:38,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:38,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:38,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:38,901 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:38,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:38,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:38,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,902 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,903 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,904 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,904 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:38,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:38,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:38,905 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:38,905 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:38,906 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,906 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:38,906 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:38,907 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:38,907 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:38,907 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:38,907 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,907 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:38,907 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:38,908 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,908 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,909 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:38,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:38,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524a274f[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:38,911 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:38,911 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,915 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:38,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:38,915 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:38,915 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,916 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:38,918 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,918 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:38,918 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,918 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:38,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:38,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:38,921 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:38,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:38,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:38,921 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:38,921 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:38,924 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,924 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,924 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:38,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:38,925 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,926 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:38,929 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:38,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:38,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:38,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:38,929 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:38,930 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:38,930 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:38,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:38,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:39,030 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:39,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:39,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:39,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:39,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:39,033 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:39,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:39,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:39,033 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:39,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,034 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,035 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:39,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:39,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:39,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,035 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:39,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:39,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:39,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:39,037 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,037 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:39,037 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:39,037 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:39,043 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:39,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:39,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:39,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20982253[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:39,045 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:39,045 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,046 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,046 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:39,046 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:39,046 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:39,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,047 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:39,048 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:39,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:39,050 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:39,051 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:39,051 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:39,051 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:39,051 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:39,053 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,053 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:39,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,053 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:39,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:39,054 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,056 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,058 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:39,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:39,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:39,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:39,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:39,058 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,058 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:39,059 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:39,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:39,062 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=633, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/78afdfe8fd1146b6a2e9525755bf2668 2023-03-19 09:56:39,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/1b9b4ade783142099e4264672a68920c 2023-03-19 09:56:39,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:39,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 530 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219859079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b0e2ddd402ea4eb8b62207fff78d9611 2023-03-19 09:56:39,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/78afdfe8fd1146b6a2e9525755bf2668 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/78afdfe8fd1146b6a2e9525755bf2668 2023-03-19 09:56:39,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/78afdfe8fd1146b6a2e9525755bf2668, entries=250, sequenceid=633, filesize=39.2 K 2023-03-19 09:56:39,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/1b9b4ade783142099e4264672a68920c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1b9b4ade783142099e4264672a68920c 2023-03-19 09:56:39,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1b9b4ade783142099e4264672a68920c, entries=150, sequenceid=633, filesize=11.9 K 2023-03-19 09:56:39,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/b0e2ddd402ea4eb8b62207fff78d9611 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b0e2ddd402ea4eb8b62207fff78d9611 2023-03-19 09:56:39,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b0e2ddd402ea4eb8b62207fff78d9611, entries=150, sequenceid=633, filesize=11.9 K 2023-03-19 09:56:39,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a391bedbca383dedc4bc8fc94dd92901 in 477ms, sequenceid=633, compaction requested=true 2023-03-19 09:56:39,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:39,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:39,105 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:39,105 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:39,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:39,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:39,105 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93176 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:39,106 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:39,106 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:39,106 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:39,106 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,106 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,106 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/494c4c14fd8547d18fcac6d872ab23c2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b8a445c932c54ca2a57af08007b0901d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/78afdfe8fd1146b6a2e9525755bf2668] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=91.0 K 2023-03-19 09:56:39,106 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9c1bb2cc6e8d4a27afdf5f2ec207274b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7105a550aec44a0b7b88135c2c292fb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1b9b4ade783142099e4264672a68920c] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.7 K 2023-03-19 09:56:39,107 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 494c4c14fd8547d18fcac6d872ab23c2, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679219797231 2023-03-19 09:56:39,107 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting b8a445c932c54ca2a57af08007b0901d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1679219797399 2023-03-19 09:56:39,107 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9c1bb2cc6e8d4a27afdf5f2ec207274b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679219797231 2023-03-19 09:56:39,108 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 78afdfe8fd1146b6a2e9525755bf2668, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679219798583 2023-03-19 09:56:39,108 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d7105a550aec44a0b7b88135c2c292fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1679219797399 2023-03-19 09:56:39,109 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1b9b4ade783142099e4264672a68920c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679219798613 2023-03-19 09:56:39,117 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#614 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:39,125 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#615 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:39,148 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/94578b33fabf412786caf10100dff648 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/94578b33fabf412786caf10100dff648 2023-03-19 09:56:39,154 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 94578b33fabf412786caf10100dff648(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:39,154 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:39,154 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=13, startTime=1679219799105; duration=0sec 2023-03-19 09:56:39,154 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:39,154 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:39,155 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:39,155 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:39,155 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,155 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1bdf55b990eb405f9c103655b8886cc5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/13a0cd3db77f4b5fbb717f2f00711e4b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b0e2ddd402ea4eb8b62207fff78d9611] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.7 K 2023-03-19 09:56:39,155 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1bdf55b990eb405f9c103655b8886cc5, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679219797231 2023-03-19 09:56:39,155 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 13a0cd3db77f4b5fbb717f2f00711e4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1679219797399 2023-03-19 09:56:39,156 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting b0e2ddd402ea4eb8b62207fff78d9611, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679219798613 2023-03-19 09:56:39,159 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:39,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:39,161 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#616 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:39,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:39,172 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:39,172 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:39,172 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:39,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:39,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,174 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:39,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:39,174 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,175 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,175 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:39,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:39,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:39,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:39,192 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-19 09:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:39,192 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:39,193 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:39,230 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303193ee3684975ba429687948197d78c513b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193ee3684975ba429687948197d78c513b 2023-03-19 09:56:39,247 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=672, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/442024608dc14c0082e06e9dad441dcc 2023-03-19 09:56:39,266 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/1bb34a5060ec429581b7c466d9e67f0c 2023-03-19 09:56:39,285 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/9cb474b3286e4330a4a7eaec8fa9cadc 2023-03-19 09:56:39,289 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/442024608dc14c0082e06e9dad441dcc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/442024608dc14c0082e06e9dad441dcc 2023-03-19 09:56:39,293 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/442024608dc14c0082e06e9dad441dcc, entries=150, sequenceid=672, filesize=25.4 K 2023-03-19 09:56:39,294 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/1bb34a5060ec429581b7c466d9e67f0c as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1bb34a5060ec429581b7c466d9e67f0c 2023-03-19 09:56:39,300 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1bb34a5060ec429581b7c466d9e67f0c, entries=150, sequenceid=672, filesize=11.9 K 2023-03-19 09:56:39,300 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-19 09:56:39,300 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/9cb474b3286e4330a4a7eaec8fa9cadc as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9cb474b3286e4330a4a7eaec8fa9cadc 2023-03-19 09:56:39,305 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9cb474b3286e4330a4a7eaec8fa9cadc, entries=150, sequenceid=672, filesize=11.9 K 2023-03-19 09:56:39,306 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for a391bedbca383dedc4bc8fc94dd92901 in 114ms, sequenceid=672, compaction requested=false 2023-03-19 09:56:39,306 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:39,306 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:39,308 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,308 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:39,308 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,308 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:39,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:39,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd98dca[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:39,310 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:39,310 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,312 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:39,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:39,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:39,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,312 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:39,313 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:39,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:39,316 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:39,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:39,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:39,316 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:39,316 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:39,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,318 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:39,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:39,319 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,320 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,323 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:39,323 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:39,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:39,323 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,323 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:39,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:39,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:39,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:39,323 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:39,324 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:39,323 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:39,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:39,324 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,424 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:39,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:39,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:39,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:39,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:39,427 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:39,427 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:39,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:39,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:39,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,428 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,432 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,432 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:39,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:39,433 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:39,433 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:39,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:39,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:39,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:39,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:39,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:39,435 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-19 09:56:39,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:39,435 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:39,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:39,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:39,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:39,436 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:39,454 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319ba5e31cd99aa4a2c955cd2ceb76b33a2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ba5e31cd99aa4a2c955cd2ceb76b33a2 2023-03-19 09:56:39,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:39,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:39,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219799494, max timestamp=1679219799503Num uniques 859; 2023-03-19 09:56:39,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219799494, max timestamp=1679219799503Num uniques 859; 2023-03-19 09:56:39,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:39,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679219799494, max timestamp=1679219799503Num uniques 859; 2023-03-19 09:56:39,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:39,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-19 09:56:39,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219799542, max timestamp=1679219799563Num uniques 1018; 2023-03-19 09:56:39,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-19 09:56:39,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219799542, max timestamp=1679219799563Num uniques 1018; 2023-03-19 09:56:39,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-19 09:56:39,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679219799542, max timestamp=1679219799563Num uniques 1018; 2023-03-19 09:56:39,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-19 09:56:39,571 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/09d4e10b4ce04c5592c12e8d2c5bb19f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09d4e10b4ce04c5592c12e8d2c5bb19f 2023-03-19 09:56:39,576 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 09d4e10b4ce04c5592c12e8d2c5bb19f(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:39,576 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:39,576 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=13, startTime=1679219799105; duration=0sec 2023-03-19 09:56:39,576 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:39,585 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/9e8db2f49fc04fc6ae61332f17360980 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9e8db2f49fc04fc6ae61332f17360980 2023-03-19 09:56:39,589 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into 9e8db2f49fc04fc6ae61332f17360980(size=26.5 K), total size for store is 51.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:39,589 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:39,589 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=13, startTime=1679219799104; duration=0sec 2023-03-19 09:56:39,589 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:39,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219799619, max timestamp=1679219799631Num uniques 1027; 2023-03-19 09:56:39,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-19 09:56:39,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219799619, max timestamp=1679219799631Num uniques 1027; 2023-03-19 09:56:39,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-19 09:56:39,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679219799619, max timestamp=1679219799631Num uniques 1027; 2023-03-19 09:56:39,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-19 09:56:39,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219799674, max timestamp=1679219799683Num uniques 1036; 2023-03-19 09:56:39,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-19 09:56:39,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219799674, max timestamp=1679219799683Num uniques 1036; 2023-03-19 09:56:39,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-19 09:56:39,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679219799674, max timestamp=1679219799683Num uniques 1036; 2023-03-19 09:56:39,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-19 09:56:39,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679219799700, max timestamp=1679219799731Num uniques 1045; 2023-03-19 09:56:39,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-19 09:56:39,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679219799700, max timestamp=1679219799731Num uniques 1045; 2023-03-19 09:56:39,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-19 09:56:39,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679219799700, max timestamp=1679219799731Num uniques 1045; 2023-03-19 09:56:39,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-19 09:56:39,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219799774, max timestamp=1679219799788Num uniques 1054; 2023-03-19 09:56:39,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219799774, max timestamp=1679219799788Num uniques 1054; 2023-03-19 09:56:39,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-19 09:56:39,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-19 09:56:39,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679219799774, max timestamp=1679219799788Num uniques 1054; 2023-03-19 09:56:39,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-19 09:56:39,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679219799828, max timestamp=1679219799837Num uniques 1013; 2023-03-19 09:56:39,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679219799828, max timestamp=1679219799837Num uniques 1013; 2023-03-19 09:56:39,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679219799828, max timestamp=1679219799837Num uniques 1013; 2023-03-19 09:56:39,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-19 09:56:39,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-19 09:56:39,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-19 09:56:39,875 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-19 09:56:39,885 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=677, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/939d3a75ad0a48e98e64751248ff3cb6 2023-03-19 09:56:39,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679219799871, max timestamp=1679219799898Num uniques 1022; 2023-03-19 09:56:39,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-19 09:56:39,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679219799871, max timestamp=1679219799898Num uniques 1022; 2023-03-19 09:56:39,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-19 09:56:39,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679219799871, max timestamp=1679219799898Num uniques 1022; 2023-03-19 09:56:39,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-19 09:56:39,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679219799949, max timestamp=1679219799962Num uniques 1031; 2023-03-19 09:56:39,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-19 09:56:39,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679219799949, max timestamp=1679219799962Num uniques 1031; 2023-03-19 09:56:39,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-19 09:56:39,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679219799949, max timestamp=1679219799962Num uniques 1031; 2023-03-19 09:56:39,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-19 09:56:40,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679219800004, max timestamp=1679219800017Num uniques 1040; 2023-03-19 09:56:40,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-19 09:56:40,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679219800004, max timestamp=1679219800017Num uniques 1040; 2023-03-19 09:56:40,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-19 09:56:40,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679219800004, max timestamp=1679219800017Num uniques 1040; 2023-03-19 09:56:40,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-19 09:56:40,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679219800076, max timestamp=1679219800094Num uniques 1049; 2023-03-19 09:56:40,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-19 09:56:40,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679219800076, max timestamp=1679219800094Num uniques 1049; 2023-03-19 09:56:40,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-19 09:56:40,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679219800076, max timestamp=1679219800094Num uniques 1049; 2023-03-19 09:56:40,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-19 09:56:40,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679219800144, max timestamp=1679219800164Num uniques 1058; 2023-03-19 09:56:40,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-19 09:56:40,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679219800144, max timestamp=1679219800164Num uniques 1058; 2023-03-19 09:56:40,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-19 09:56:40,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679219800144, max timestamp=1679219800164Num uniques 1058; 2023-03-19 09:56:40,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-19 09:56:40,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679219800208, max timestamp=1679219800218Num uniques 1017; 2023-03-19 09:56:40,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-19 09:56:40,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679219800208, max timestamp=1679219800218Num uniques 1017; 2023-03-19 09:56:40,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-19 09:56:40,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679219800208, max timestamp=1679219800218Num uniques 1017; 2023-03-19 09:56:40,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-19 09:56:40,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679219800232, max timestamp=1679219800279Num uniques 1026; 2023-03-19 09:56:40,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-19 09:56:40,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679219800232, max timestamp=1679219800279Num uniques 1026; 2023-03-19 09:56:40,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-19 09:56:40,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679219800232, max timestamp=1679219800279Num uniques 1026; 2023-03-19 09:56:40,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-19 09:56:40,305 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/300c874367b54934a7d722a956235c98 2023-03-19 09:56:40,314 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/cffe47cc9a5042a3b36d2d94a112ebb9 2023-03-19 09:56:40,317 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/939d3a75ad0a48e98e64751248ff3cb6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/939d3a75ad0a48e98e64751248ff3cb6 2023-03-19 09:56:40,320 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/939d3a75ad0a48e98e64751248ff3cb6, entries=100, sequenceid=677, filesize=18.6 K 2023-03-19 09:56:40,321 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/300c874367b54934a7d722a956235c98 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/300c874367b54934a7d722a956235c98 2023-03-19 09:56:40,325 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/300c874367b54934a7d722a956235c98, entries=50, sequenceid=677, filesize=7.1 K 2023-03-19 09:56:40,325 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/cffe47cc9a5042a3b36d2d94a112ebb9 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cffe47cc9a5042a3b36d2d94a112ebb9 2023-03-19 09:56:40,332 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cffe47cc9a5042a3b36d2d94a112ebb9, entries=50, sequenceid=677, filesize=7.1 K 2023-03-19 09:56:40,332 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=107.34 KB/109920 for a391bedbca383dedc4bc8fc94dd92901 in 897ms, sequenceid=677, compaction requested=true 2023-03-19 09:56:40,332 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:40,332 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:40,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=114.05 KB heapSize=323.63 KB 2023-03-19 09:56:40,334 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,335 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,335 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:40,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34494ccc[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:40,337 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:40,337 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,338 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:40,338 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:40,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,339 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:40,341 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:40,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:40,344 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:40,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:40,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:40,344 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:40,344 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:40,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,347 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,348 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,362 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:40,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:40,365 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,366 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:40,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,366 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,366 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:40,366 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:40,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:40,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 798 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219860386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,466 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:40,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:40,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:40,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:40,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:40,471 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:40,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:40,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:40,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,471 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,472 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,472 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,473 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,473 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:40,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:40,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:40,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:40,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:40,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,484 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:40,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:40,488 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:40,488 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:40,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:40,489 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,489 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:40,489 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:40,489 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,489 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:40,489 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:40,490 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,491 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,491 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:40,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@373a6555[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:40,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:40,492 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:40,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 800 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219860490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,495 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,495 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:40,495 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:40,495 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:40,495 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:40,500 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:40,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,500 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:40,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:40,502 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:40,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:40,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:40,502 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:40,502 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:40,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,504 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,504 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,506 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,508 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:40,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:40,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:40,508 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,509 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,509 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:40,509 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:40,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,609 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:40,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:40,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:40,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:40,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:40,612 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:40,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:40,612 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:40,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,613 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,615 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,615 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,616 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:40,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:40,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:40,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:40,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:40,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:40,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:40,617 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:40,617 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,617 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:40,617 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:40,617 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:40,618 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:40,619 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,619 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:40,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d1535b[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:40,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:40,621 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,622 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,622 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:40,622 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:40,623 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:40,623 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:40,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:40,624 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,624 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,625 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:40,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:40,627 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:40,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:40,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:40,627 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:40,627 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:40,634 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,634 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,634 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,635 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,636 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,639 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:40,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:40,639 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,640 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,640 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:40,640 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:40,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:40,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:40,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 802 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219860693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,740 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:40,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:40,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:40,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:40,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:40,743 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:40,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:40,743 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:40,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,744 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,745 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,745 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,746 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,746 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:40,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:40,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:40,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:40,747 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,747 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:40,747 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:40,747 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:40,747 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:40,749 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,750 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:40,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a63396[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:40,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:40,751 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,753 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:40,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:40,753 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:40,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,754 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:40,755 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:40,755 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,755 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:40,755 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,758 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319ea388d72621941078745c74836c2b22a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ea388d72621941078745c74836c2b22a 2023-03-19 09:56:40,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:40,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:40,760 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:40,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:40,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:40,760 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:40,760 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:40,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,762 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,763 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,764 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,766 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=936, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/1e68e70bd04a4fadb9248b1858cb5ba6 2023-03-19 09:56:40,766 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,766 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,767 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,767 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,767 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,767 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,767 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,767 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,767 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:40,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:40,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,768 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:40,768 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:40,767 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:40,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/af253bcc668f4d0eb35c524d0663127a 2023-03-19 09:56:40,868 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:40,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:40,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:40,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:40,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:40,871 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:40,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:40,871 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:40,872 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,872 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,874 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,874 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,875 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,875 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:40,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:40,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:40,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:40,876 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,876 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:40,876 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:40,876 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:40,876 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:40,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:40,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:40,877 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:40,878 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,878 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,879 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f722a2[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:40,880 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:40,881 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,882 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:40,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:40,882 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:40,882 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,882 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:40,886 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:40,886 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,886 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,886 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:40,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:40,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:40,889 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:40,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:40,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:40,890 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:40,890 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:40,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,897 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:40,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,898 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,899 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,902 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,902 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,902 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:40,902 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:40,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:40,902 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:40,903 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:40,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:40,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:40,903 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:40,903 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,903 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:40,903 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:40,903 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:40,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:40,903 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:41,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:41,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 804 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219860999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,003 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:41,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:41,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:41,007 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:41,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:41,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:41,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,007 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,008 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:41,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:41,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,009 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:41,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:41,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:41,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:41,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:41,010 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:41,010 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:41,011 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,011 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:41,011 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:41,011 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:41,011 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,018 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,019 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:41,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c03e224[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:41,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:41,021 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,022 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:41,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:41,022 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:41,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,023 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:41,024 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:41,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:41,028 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:41,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:41,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:41,028 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:41,028 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:41,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,030 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:41,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,031 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,032 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:41,036 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:41,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:41,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:41,036 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,037 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:41,037 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:41,037 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:41,153 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:41,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:41,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:41,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:41,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:41,158 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:41,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:41,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:41,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,158 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,159 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:41,160 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:41,160 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,160 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:41,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:41,161 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,164 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:41,164 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:41,164 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,164 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:41,164 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:41,164 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,164 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:41,164 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:41,165 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,165 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,166 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:41,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50942374[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:41,167 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:41,167 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,168 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:41,169 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,169 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:41,171 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:41,173 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:41,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:41,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:41,173 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:41,173 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:41,175 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,175 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:41,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,175 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,176 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,177 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,180 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:41,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,180 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,181 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:41,181 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:41,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:41,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:41,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/53e47021cf734091807b7218e214fc1b 2023-03-19 09:56:41,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/1e68e70bd04a4fadb9248b1858cb5ba6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1e68e70bd04a4fadb9248b1858cb5ba6 2023-03-19 09:56:41,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1e68e70bd04a4fadb9248b1858cb5ba6, entries=200, sequenceid=936, filesize=32.3 K 2023-03-19 09:56:41,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/af253bcc668f4d0eb35c524d0663127a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/af253bcc668f4d0eb35c524d0663127a 2023-03-19 09:56:41,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/af253bcc668f4d0eb35c524d0663127a, entries=150, sequenceid=936, filesize=11.9 K 2023-03-19 09:56:41,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/53e47021cf734091807b7218e214fc1b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/53e47021cf734091807b7218e214fc1b 2023-03-19 09:56:41,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/53e47021cf734091807b7218e214fc1b, entries=150, sequenceid=936, filesize=11.9 K 2023-03-19 09:56:41,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~323.58 KB/331344, currentSize=67.09 KB/68700 for a391bedbca383dedc4bc8fc94dd92901 in 885ms, sequenceid=936, compaction requested=true 2023-03-19 09:56:41,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:41,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:41,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:41,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:41,219 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:41,219 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:41,220 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105230 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:41,220 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44948 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:41,220 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:41,220 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:41,220 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,220 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,220 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9e8db2f49fc04fc6ae61332f17360980, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/442024608dc14c0082e06e9dad441dcc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/939d3a75ad0a48e98e64751248ff3cb6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1e68e70bd04a4fadb9248b1858cb5ba6] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=102.8 K 2023-03-19 09:56:41,220 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/94578b33fabf412786caf10100dff648, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1bb34a5060ec429581b7c466d9e67f0c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/300c874367b54934a7d722a956235c98, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/af253bcc668f4d0eb35c524d0663127a] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=43.9 K 2023-03-19 09:56:41,221 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9e8db2f49fc04fc6ae61332f17360980, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679219798613 2023-03-19 09:56:41,221 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 94578b33fabf412786caf10100dff648, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679219798613 2023-03-19 09:56:41,221 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 442024608dc14c0082e06e9dad441dcc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679219798761 2023-03-19 09:56:41,221 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 1bb34a5060ec429581b7c466d9e67f0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679219798761 2023-03-19 09:56:41,222 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 939d3a75ad0a48e98e64751248ff3cb6, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679219799402 2023-03-19 09:56:41,222 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 300c874367b54934a7d722a956235c98, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679219799423 2023-03-19 09:56:41,222 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 1e68e70bd04a4fadb9248b1858cb5ba6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1679219800316 2023-03-19 09:56:41,222 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting af253bcc668f4d0eb35c524d0663127a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1679219800324 2023-03-19 09:56:41,229 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#627 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:41,233 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#626 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:41,260 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/0a74d8b066094bc39278c06766ebe0c3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/0a74d8b066094bc39278c06766ebe0c3 2023-03-19 09:56:41,265 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 0a74d8b066094bc39278c06766ebe0c3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:41,265 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:41,265 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=12, startTime=1679219801219; duration=0sec 2023-03-19 09:56:41,265 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:41,265 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-19 09:56:41,266 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44948 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-19 09:56:41,266 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:41,266 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,266 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09d4e10b4ce04c5592c12e8d2c5bb19f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9cb474b3286e4330a4a7eaec8fa9cadc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cffe47cc9a5042a3b36d2d94a112ebb9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/53e47021cf734091807b7218e214fc1b] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=43.9 K 2023-03-19 09:56:41,267 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 09d4e10b4ce04c5592c12e8d2c5bb19f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679219798613 2023-03-19 09:56:41,267 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 9cb474b3286e4330a4a7eaec8fa9cadc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679219798761 2023-03-19 09:56:41,268 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting cffe47cc9a5042a3b36d2d94a112ebb9, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679219799423 2023-03-19 09:56:41,268 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 53e47021cf734091807b7218e214fc1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1679219800324 2023-03-19 09:56:41,277 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#628 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:41,281 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:41,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:41,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:41,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:41,288 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:41,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:41,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:41,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,288 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,290 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:41,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,290 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:41,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,291 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:41,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:41,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:41,307 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:41,307 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,307 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:41,307 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:41,307 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:41,307 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:41,308 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-19 09:56:41,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:41,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:41,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:41,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,337 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319145df27b07004cb4a10efa9c2deaceb8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319145df27b07004cb4a10efa9c2deaceb8 2023-03-19 09:56:41,346 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=950, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/9109fc85e0144fa6be9f5b50fed09c40 2023-03-19 09:56:41,370 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=950 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/919a9a7f0127484498ff3eca8395c303 2023-03-19 09:56:41,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:41,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:41,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:41,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 822 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219861569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:41,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 824 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219861673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,690 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/512d625acee749ecaad837d775b08c4f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/512d625acee749ecaad837d775b08c4f 2023-03-19 09:56:41,694 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into 512d625acee749ecaad837d775b08c4f(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:41,694 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:41,694 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=12, startTime=1679219801219; duration=0sec 2023-03-19 09:56:41,694 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:41,701 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/567b81beb35b4aaa90d237f2f35036db as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/567b81beb35b4aaa90d237f2f35036db 2023-03-19 09:56:41,705 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 567b81beb35b4aaa90d237f2f35036db(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:41,706 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:41,706 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=12, startTime=1679219801219; duration=0sec 2023-03-19 09:56:41,706 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:41,806 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=950 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/a2fcbab69a5a42328ab74ca0ca334da2 2023-03-19 09:56:41,810 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/9109fc85e0144fa6be9f5b50fed09c40 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9109fc85e0144fa6be9f5b50fed09c40 2023-03-19 09:56:41,813 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9109fc85e0144fa6be9f5b50fed09c40, entries=150, sequenceid=950, filesize=25.4 K 2023-03-19 09:56:41,814 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/919a9a7f0127484498ff3eca8395c303 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/919a9a7f0127484498ff3eca8395c303 2023-03-19 09:56:41,818 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/919a9a7f0127484498ff3eca8395c303, entries=150, sequenceid=950, filesize=11.9 K 2023-03-19 09:56:41,819 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/a2fcbab69a5a42328ab74ca0ca334da2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a2fcbab69a5a42328ab74ca0ca334da2 2023-03-19 09:56:41,825 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a2fcbab69a5a42328ab74ca0ca334da2, entries=150, sequenceid=950, filesize=11.9 K 2023-03-19 09:56:41,826 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a391bedbca383dedc4bc8fc94dd92901 in 518ms, sequenceid=950, compaction requested=false 2023-03-19 09:56:41,826 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:41,826 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:41,826 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:41,829 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,829 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:41,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77098073[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:41,832 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:41,832 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,833 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:41,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,834 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:41,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:41,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:41,835 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:41,835 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:41,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:41,839 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:41,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:41,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:41,839 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:41,840 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:41,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:41,841 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:41,842 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,844 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,852 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:41,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:41,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:41,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:41,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:41,853 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:41,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:41,853 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:41,853 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:41,853 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:41,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:41,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:41,879 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-19 09:56:41,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:41,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:41,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:41,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,891 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319be500c48dfc747ef90c9da8629186bcd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319be500c48dfc747ef90c9da8629186bcd 2023-03-19 09:56:41,900 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=972, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/c82f08c1e5d74a0c800ef97acd452702 2023-03-19 09:56:41,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:41,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=972 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/55968ac0f4ca4c168f2cac941e9c3414 2023-03-19 09:56:41,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 836 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219861913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=972 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/0deec4409bc945d0a99485efb16d312d 2023-03-19 09:56:41,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/c82f08c1e5d74a0c800ef97acd452702 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/c82f08c1e5d74a0c800ef97acd452702 2023-03-19 09:56:41,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/c82f08c1e5d74a0c800ef97acd452702, entries=200, sequenceid=972, filesize=32.3 K 2023-03-19 09:56:41,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/55968ac0f4ca4c168f2cac941e9c3414 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55968ac0f4ca4c168f2cac941e9c3414 2023-03-19 09:56:41,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55968ac0f4ca4c168f2cac941e9c3414, entries=150, sequenceid=972, filesize=11.9 K 2023-03-19 09:56:41,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/0deec4409bc945d0a99485efb16d312d as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0deec4409bc945d0a99485efb16d312d 2023-03-19 09:56:41,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0deec4409bc945d0a99485efb16d312d, entries=150, sequenceid=972, filesize=11.9 K 2023-03-19 09:56:41,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a391bedbca383dedc4bc8fc94dd92901 in 63ms, sequenceid=972, compaction requested=true 2023-03-19 09:56:41,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:41,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:41,942 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:41,942 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:41,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:41,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:41,943 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86370 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:41,943 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:41,944 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,944 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/512d625acee749ecaad837d775b08c4f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9109fc85e0144fa6be9f5b50fed09c40, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/c82f08c1e5d74a0c800ef97acd452702] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=84.3 K 2023-03-19 09:56:41,944 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:41,944 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:41,944 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,944 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/0a74d8b066094bc39278c06766ebe0c3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/919a9a7f0127484498ff3eca8395c303, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55968ac0f4ca4c168f2cac941e9c3414] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.9 K 2023-03-19 09:56:41,945 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 512d625acee749ecaad837d775b08c4f, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1679219800324 2023-03-19 09:56:41,945 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 9109fc85e0144fa6be9f5b50fed09c40, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=950, earliestPutTs=1679219800371 2023-03-19 09:56:41,945 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0a74d8b066094bc39278c06766ebe0c3, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1679219800324 2023-03-19 09:56:41,945 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting c82f08c1e5d74a0c800ef97acd452702, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1679219801553 2023-03-19 09:56:41,945 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 919a9a7f0127484498ff3eca8395c303, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=950, earliestPutTs=1679219800371 2023-03-19 09:56:41,946 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 55968ac0f4ca4c168f2cac941e9c3414, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1679219801559 2023-03-19 09:56:41,952 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#636 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:41,953 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:41,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:41,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-19 09:56:41,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-19 09:56:41,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-19 09:56:41,974 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-19 09:56:41,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-19 09:56:41,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-19 09:56:41,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,974 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,976 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#635 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:41,977 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:41,977 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:41,978 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-19 09:56:41,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-19 09:56:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:41,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-19 09:56:41,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-19 09:56:41,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-19 09:56:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-19 09:56:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-19 09:56:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-19 09:56:41,992 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/7cb357bfdb5b4218a58ac0a61cd4a0a6 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/7cb357bfdb5b4218a58ac0a61cd4a0a6 2023-03-19 09:56:41,997 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into 7cb357bfdb5b4218a58ac0a61cd4a0a6(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:41,997 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:41,997 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=13, startTime=1679219801942; duration=0sec 2023-03-19 09:56:41,997 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:41,997 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:41,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-19 09:56:41,998 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,998 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. started... 2023-03-19 09:56:41,998 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-19 09:56:41,998 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:41,998 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-19 09:56:41,998 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:41,998 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:41,998 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:41,998 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/567b81beb35b4aaa90d237f2f35036db, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a2fcbab69a5a42328ab74ca0ca334da2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0deec4409bc945d0a99485efb16d312d] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=36.9 K 2023-03-19 09:56:41,999 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,999 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:41,999 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,999 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:41,999 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:41,999 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 567b81beb35b4aaa90d237f2f35036db, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1679219800324 2023-03-19 09:56:41,999 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting a2fcbab69a5a42328ab74ca0ca334da2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=950, earliestPutTs=1679219800371 2023-03-19 09:56:42,000 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 0deec4409bc945d0a99485efb16d312d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1679219801559 2023-03-19 09:56:42,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. as already flushing 2023-03-19 09:56:42,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:42,025 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#637 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:42,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:42,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34669] ipc.CallRunner(144): callId: 855 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219862082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:42,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34669] ipc.CallRunner(144): callId: 857 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219862186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:42,240 DEBUG [Thread-3962] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:54245 2023-03-19 09:56:42,240 DEBUG [Thread-3962] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:42,242 DEBUG [Thread-3964] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:54245 2023-03-19 09:56:42,242 DEBUG [Thread-3964] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:42,243 DEBUG [Thread-3968] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:54245 2023-03-19 09:56:42,243 DEBUG [Thread-3968] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:42,243 DEBUG [Thread-3970] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:54245 2023-03-19 09:56:42,244 DEBUG [Thread-3970] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:42,245 DEBUG [Thread-3966] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:54245 2023-03-19 09:56:42,245 DEBUG [Thread-3966] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:42,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:42,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 859 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219862391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:42,462 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/130a18271d164ddeb486691a7b54035a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/130a18271d164ddeb486691a7b54035a 2023-03-19 09:56:42,466 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/d73b8a043c6a49d389e1c5d8c6cbefb3 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/d73b8a043c6a49d389e1c5d8c6cbefb3 2023-03-19 09:56:42,466 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 130a18271d164ddeb486691a7b54035a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:42,466 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:42,466 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=13, startTime=1679219801943; duration=0sec 2023-03-19 09:56:42,467 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:42,469 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into d73b8a043c6a49d389e1c5d8c6cbefb3(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:42,469 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:42,470 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=13, startTime=1679219801942; duration=0sec 2023-03-19 09:56:42,470 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:42,476 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319ace95c0e362c48ff9701c1de4401dfd3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ace95c0e362c48ff9701c1de4401dfd3 2023-03-19 09:56:42,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:42,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 861 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219862694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:42,883 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=985, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/67e82bc6e5fc415086ddfa5a597b935a 2023-03-19 09:56:42,890 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/69bdc3365a7f40a781ac6219a4a89d07 2023-03-19 09:56:43,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-19 09:56:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] ipc.CallRunner(144): callId: 863 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36762 deadline: 1679219863199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,298 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/3d949a6fa2ed466bb9b49f01189a03ac 2023-03-19 09:56:43,302 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/67e82bc6e5fc415086ddfa5a597b935a as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/67e82bc6e5fc415086ddfa5a597b935a 2023-03-19 09:56:43,304 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/67e82bc6e5fc415086ddfa5a597b935a, entries=150, sequenceid=985, filesize=25.4 K 2023-03-19 09:56:43,305 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/69bdc3365a7f40a781ac6219a4a89d07 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/69bdc3365a7f40a781ac6219a4a89d07 2023-03-19 09:56:43,308 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/69bdc3365a7f40a781ac6219a4a89d07, entries=150, sequenceid=985, filesize=11.9 K 2023-03-19 09:56:43,308 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/3d949a6fa2ed466bb9b49f01189a03ac as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3d949a6fa2ed466bb9b49f01189a03ac 2023-03-19 09:56:43,311 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3d949a6fa2ed466bb9b49f01189a03ac, entries=150, sequenceid=985, filesize=11.9 K 2023-03-19 09:56:43,312 INFO [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a391bedbca383dedc4bc8fc94dd92901 in 1314ms, sequenceid=985, compaction requested=false 2023-03-19 09:56:43,312 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:43,312 DEBUG [rs(jenkins-hbase4.apache.org,34669,1679219618625)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:43,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-19 09:56:43,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-19 09:56:43,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-19 09:56:43,312 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-19 09:56:43,313 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,313 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:43,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,314 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-19 09:56:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:43,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34669,1679219618625' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-19 09:56:43,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ba5e02[Count = 0] remaining members to acquire global barrier 2023-03-19 09:56:43,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-19 09:56:43,315 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,317 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-19 09:56:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-19 09:56:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34669,1679219618625' in zk 2023-03-19 09:56:43,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,318 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-19 09:56:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-19 09:56:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:43,319 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,34669,1679219618625' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-19 09:56:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34669,1679219618625': 2023-03-19 09:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34669,1679219618625' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-19 09:56:43,321 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-19 09:56:43,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-19 09:56:43,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-19 09:56:43,321 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-19 09:56:43,321 INFO [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-19 09:56:43,322 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-19 09:56:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-19 09:56:43,322 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:43,323 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:43,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:43,323 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-19 09:56:43,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-19 09:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:43,324 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-19 09:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-19 09:56:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-19 09:56:43,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-19 09:56:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-19 09:56:43,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:43,326 DEBUG [(jenkins-hbase4.apache.org,44865,1679219617257)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-19 09:56:43,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-19 09:56:43,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-19 09:56:43,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-19 09:56:43,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-19 09:56:43,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-19 09:56:43,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-19 09:56:43,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-19 09:56:43,326 DEBUG [Thread-3961] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-19 09:56:43,326 DEBUG [Thread-3961] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-19 09:56:43,326 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,327 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-19 09:56:43,327 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-19 09:56:43,427 DEBUG [Thread-3961] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-19 09:56:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-19 09:56:44,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34669] regionserver.HRegion(9158): Flush requested on a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:44,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-19 09:56:44,202 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:44,202 DEBUG [Thread-3955] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:54245 2023-03-19 09:56:44,202 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:44,203 DEBUG [Thread-3955] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:44,203 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:44,203 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:44,203 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:44,203 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-19 09:56:44,613 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230319aadd75c552f245e1b743fa974a136d8c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319aadd75c552f245e1b743fa974a136d8c 2023-03-19 09:56:45,017 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1008, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/956fd2e572864dce86cb7c1bc264d95b 2023-03-19 09:56:45,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1008 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/d7386f7f31e64291b06d00f7b9149996 2023-03-19 09:56:45,743 DEBUG [Thread-3953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:54245 2023-03-19 09:56:45,743 DEBUG [Thread-3953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:45,743 DEBUG [Thread-3959] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:54245 2023-03-19 09:56:45,743 DEBUG [Thread-3959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:45,753 DEBUG [Thread-3957] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:54245 2023-03-19 09:56:45,753 DEBUG [Thread-3957] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:45,812 DEBUG [Thread-3951] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:54245 2023-03-19 09:56:45,812 DEBUG [Thread-3951] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:45,812 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-19 09:56:45,812 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-03-19 09:56:45,812 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-03-19 09:56:45,812 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 730 2023-03-19 09:56:45,812 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 3635 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 3660 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 3611 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 3627 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(398): read 3689 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-19 09:56:45,813 INFO [Listener at localhost/45961] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-19 09:56:45,813 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68067de1 to 127.0.0.1:54245 2023-03-19 09:56:45,813 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:45,814 INFO [Listener at localhost/45961] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-19 09:56:45,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-19 09:56:45,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-19 09:56:45,817 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219805817"}]},"ts":"1679219805817"} 2023-03-19 09:56:45,818 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-19 09:56:45,819 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-19 09:56:45,820 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, UNASSIGN}] 2023-03-19 09:56:45,821 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, UNASSIGN 2023-03-19 09:56:45,821 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:45,821 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219805821"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679219805821"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679219805821"}]},"ts":"1679219805821"} 2023-03-19 09:56:45,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625}] 2023-03-19 09:56:45,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1008 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/97dfcd691d95452190c485aa9215aea2 2023-03-19 09:56:45,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/956fd2e572864dce86cb7c1bc264d95b as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/956fd2e572864dce86cb7c1bc264d95b 2023-03-19 09:56:45,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/956fd2e572864dce86cb7c1bc264d95b, entries=150, sequenceid=1008, filesize=25.4 K 2023-03-19 09:56:45,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/d7386f7f31e64291b06d00f7b9149996 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7386f7f31e64291b06d00f7b9149996 2023-03-19 09:56:45,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7386f7f31e64291b06d00f7b9149996, entries=150, sequenceid=1008, filesize=11.9 K 2023-03-19 09:56:45,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/97dfcd691d95452190c485aa9215aea2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/97dfcd691d95452190c485aa9215aea2 2023-03-19 09:56:45,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/97dfcd691d95452190c485aa9215aea2, entries=150, sequenceid=1008, filesize=11.9 K 2023-03-19 09:56:45,847 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=26.84 KB/27480 for a391bedbca383dedc4bc8fc94dd92901 in 1645ms, sequenceid=1008, compaction requested=true 2023-03-19 09:56:45,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:45,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:45,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:45,847 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:45,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:45,847 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:45,848 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:45,848 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79432 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:45,848 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/B is initiating minor compaction (all files) 2023-03-19 09:56:45,848 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/A is initiating minor compaction (all files) 2023-03-19 09:56:45,848 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/B in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:45,848 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/A in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:45,848 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/7cb357bfdb5b4218a58ac0a61cd4a0a6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/69bdc3365a7f40a781ac6219a4a89d07, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7386f7f31e64291b06d00f7b9149996] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=37.0 K 2023-03-19 09:56:45,848 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/d73b8a043c6a49d389e1c5d8c6cbefb3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/67e82bc6e5fc415086ddfa5a597b935a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/956fd2e572864dce86cb7c1bc264d95b] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=77.6 K 2023-03-19 09:56:45,848 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 7cb357bfdb5b4218a58ac0a61cd4a0a6, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1679219801559 2023-03-19 09:56:45,849 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting d73b8a043c6a49d389e1c5d8c6cbefb3, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1679219801559 2023-03-19 09:56:45,849 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 69bdc3365a7f40a781ac6219a4a89d07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=985, earliestPutTs=1679219801891 2023-03-19 09:56:45,849 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 67e82bc6e5fc415086ddfa5a597b935a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=985, earliestPutTs=1679219801891 2023-03-19 09:56:45,849 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting d7386f7f31e64291b06d00f7b9149996, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1008, earliestPutTs=1679219802071 2023-03-19 09:56:45,849 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] compactions.Compactor(207): Compacting 956fd2e572864dce86cb7c1bc264d95b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1008, earliestPutTs=1679219802071 2023-03-19 09:56:45,854 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#B#compaction#644 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:45,858 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#A#compaction#645 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:45,862 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e0091e3029244976b1b8b4689dc3a1a2 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0091e3029244976b1b8b4689dc3a1a2 2023-03-19 09:56:45,865 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/B of a391bedbca383dedc4bc8fc94dd92901 into e0091e3029244976b1b8b4689dc3a1a2(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:45,865 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:45,865 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/B, priority=13, startTime=1679219805847; duration=0sec 2023-03-19 09:56:45,865 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-19 09:56:45,865 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-19 09:56:45,865 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-19 09:56:45,866 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1912): a391bedbca383dedc4bc8fc94dd92901/C is initiating minor compaction (all files) 2023-03-19 09:56:45,866 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a391bedbca383dedc4bc8fc94dd92901/C in TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:45,866 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/130a18271d164ddeb486691a7b54035a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3d949a6fa2ed466bb9b49f01189a03ac, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/97dfcd691d95452190c485aa9215aea2] into tmpdir=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp, totalSize=37.0 K 2023-03-19 09:56:45,866 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 130a18271d164ddeb486691a7b54035a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=972, earliestPutTs=1679219801559 2023-03-19 09:56:45,866 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 3d949a6fa2ed466bb9b49f01189a03ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=985, earliestPutTs=1679219801891 2023-03-19 09:56:45,866 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] compactions.Compactor(207): Compacting 97dfcd691d95452190c485aa9215aea2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1008, earliestPutTs=1679219802071 2023-03-19 09:56:45,871 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] throttle.PressureAwareThroughputController(145): a391bedbca383dedc4bc8fc94dd92901#C#compaction#646 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-19 09:56:45,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-19 09:56:45,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:45,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a391bedbca383dedc4bc8fc94dd92901, disabling compactions & flushes 2023-03-19 09:56:45,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 2 compactions to complete for region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:46,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-19 09:56:46,278 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/7e6213f439b34c8f951e6693af436df7 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/7e6213f439b34c8f951e6693af436df7 2023-03-19 09:56:46,281 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/C of a391bedbca383dedc4bc8fc94dd92901 into 7e6213f439b34c8f951e6693af436df7(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-19 09:56:46,281 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:46,281 INFO [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/C, priority=13, startTime=1679219805847; duration=0sec 2023-03-19 09:56:46,281 DEBUG [RS:0;jenkins-hbase4:34669-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:46,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-19 09:56:46,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-19 09:56:47,074 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/b15e7bda18824097b401f5fc0b877c51 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b15e7bda18824097b401f5fc0b877c51 2023-03-19 09:56:47,078 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a391bedbca383dedc4bc8fc94dd92901/A of a391bedbca383dedc4bc8fc94dd92901 into b15e7bda18824097b401f5fc0b877c51(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-19 09:56:47,078 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:47,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:47,078 INFO [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901., storeName=a391bedbca383dedc4bc8fc94dd92901/A, priority=13, startTime=1679219805847; duration=1sec 2023-03-19 09:56:47,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:47,079 DEBUG [RS:0;jenkins-hbase4:34669-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. after waiting 0 ms 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:47,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a391bedbca383dedc4bc8fc94dd92901 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=A 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=B 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a391bedbca383dedc4bc8fc94dd92901, store=C 2023-03-19 09:56:47,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-19 09:56:47,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303193091bfee825f42128dacf2e6d1a9a917 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193091bfee825f42128dacf2e6d1a9a917 2023-03-19 09:56:47,893 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1018, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/f383e0d1124349aeae3db6665997139f 2023-03-19 09:56:47,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-19 09:56:48,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1018 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e7197c6e5d21452c812cf10e328d7993 2023-03-19 09:56:48,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1018 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/20433e82fc8f4ccb96c385128a4a636f 2023-03-19 09:56:48,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/A/f383e0d1124349aeae3db6665997139f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/f383e0d1124349aeae3db6665997139f 2023-03-19 09:56:48,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/f383e0d1124349aeae3db6665997139f, entries=100, sequenceid=1018, filesize=18.6 K 2023-03-19 09:56:48,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/B/e7197c6e5d21452c812cf10e328d7993 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e7197c6e5d21452c812cf10e328d7993 2023-03-19 09:56:48,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e7197c6e5d21452c812cf10e328d7993, entries=100, sequenceid=1018, filesize=9.5 K 2023-03-19 09:56:48,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/.tmp/C/20433e82fc8f4ccb96c385128a4a636f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/20433e82fc8f4ccb96c385128a4a636f 2023-03-19 09:56:48,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/20433e82fc8f4ccb96c385128a4a636f, entries=100, sequenceid=1018, filesize=9.5 K 2023-03-19 09:56:48,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a391bedbca383dedc4bc8fc94dd92901 in 1642ms, sequenceid=1018, compaction requested=false 2023-03-19 09:56:48,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/49f819207a954b3e819c5a2aecae0cc7, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/93feb0764c4a4256a2587e18bd7865c0, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5940edeb7f764922bd26b689e3de117c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/3f0906d6af6c4bac8d9ee4330237c2d3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9acbe182f9c343639ca4402b1889ff45, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7e63a3e95dc14c98b91d0ff8741dfbee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/592d75135cc64487a1e7c4bd3423c6bf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/95260dcb7c4846a18c6d842f0dd909ee, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/33e9e4dee7404f0f8d32e917adc75f85, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5572e54aa7914819a50ab32afad17041, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1a4947223de24711987dcafb8ee43efa, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0774fe9157c64e7983a63f492c44cfd3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/21a6415becf243d7a01458396db37e61, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5d507181d5a44813a752f5b48ab4c4b5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b4882f54a30f425a8d49c84533495d95, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/2ac845969a214e8482c44cbde0fa4c61, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/835840e36e5f42f49f6ee1df4888a4a5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/6c7a3ee77a7145f3827d3d880d73fd30, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/bc33dfba55f44f85819d54cc7fa57424, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9fe2909631e74e39b92351dd6caab938, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/849521f4f5ee4205870e4d9b2bc2f896, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0819bad8e6b9447e8a3abab79f5695ca, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7b4b41ef1a964af388c7dfab72d1fe85, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/930a23777965496d87c0ec251809f91b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/211e88428635494ab38026a9322b9fac, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/01fb137e97e44246ab5e8063398f3b13, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/92438a70ca2b41d2a075749d70791d19, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/e75bf80099d5417a8858e126b17685e5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/494c4c14fd8547d18fcac6d872ab23c2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b8a445c932c54ca2a57af08007b0901d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/78afdfe8fd1146b6a2e9525755bf2668, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9e8db2f49fc04fc6ae61332f17360980, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/442024608dc14c0082e06e9dad441dcc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/939d3a75ad0a48e98e64751248ff3cb6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1e68e70bd04a4fadb9248b1858cb5ba6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/512d625acee749ecaad837d775b08c4f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9109fc85e0144fa6be9f5b50fed09c40, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/c82f08c1e5d74a0c800ef97acd452702, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/d73b8a043c6a49d389e1c5d8c6cbefb3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/67e82bc6e5fc415086ddfa5a597b935a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/956fd2e572864dce86cb7c1bc264d95b] to archive 2023-03-19 09:56:48,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:56:48,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/49f819207a954b3e819c5a2aecae0cc7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/49f819207a954b3e819c5a2aecae0cc7 2023-03-19 09:56:48,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/93feb0764c4a4256a2587e18bd7865c0 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/93feb0764c4a4256a2587e18bd7865c0 2023-03-19 09:56:48,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5940edeb7f764922bd26b689e3de117c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5940edeb7f764922bd26b689e3de117c 2023-03-19 09:56:48,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/3f0906d6af6c4bac8d9ee4330237c2d3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/3f0906d6af6c4bac8d9ee4330237c2d3 2023-03-19 09:56:48,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9acbe182f9c343639ca4402b1889ff45 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9acbe182f9c343639ca4402b1889ff45 2023-03-19 09:56:48,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7e63a3e95dc14c98b91d0ff8741dfbee to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7e63a3e95dc14c98b91d0ff8741dfbee 2023-03-19 09:56:48,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/592d75135cc64487a1e7c4bd3423c6bf to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/592d75135cc64487a1e7c4bd3423c6bf 2023-03-19 09:56:48,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/95260dcb7c4846a18c6d842f0dd909ee to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/95260dcb7c4846a18c6d842f0dd909ee 2023-03-19 09:56:48,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/33e9e4dee7404f0f8d32e917adc75f85 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/33e9e4dee7404f0f8d32e917adc75f85 2023-03-19 09:56:48,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5572e54aa7914819a50ab32afad17041 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5572e54aa7914819a50ab32afad17041 2023-03-19 09:56:48,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1a4947223de24711987dcafb8ee43efa to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1a4947223de24711987dcafb8ee43efa 2023-03-19 09:56:48,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0774fe9157c64e7983a63f492c44cfd3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0774fe9157c64e7983a63f492c44cfd3 2023-03-19 09:56:48,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/21a6415becf243d7a01458396db37e61 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/21a6415becf243d7a01458396db37e61 2023-03-19 09:56:48,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5d507181d5a44813a752f5b48ab4c4b5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/5d507181d5a44813a752f5b48ab4c4b5 2023-03-19 09:56:48,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b4882f54a30f425a8d49c84533495d95 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b4882f54a30f425a8d49c84533495d95 2023-03-19 09:56:48,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/2ac845969a214e8482c44cbde0fa4c61 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/2ac845969a214e8482c44cbde0fa4c61 2023-03-19 09:56:48,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/835840e36e5f42f49f6ee1df4888a4a5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/835840e36e5f42f49f6ee1df4888a4a5 2023-03-19 09:56:48,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/6c7a3ee77a7145f3827d3d880d73fd30 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/6c7a3ee77a7145f3827d3d880d73fd30 2023-03-19 09:56:48,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/bc33dfba55f44f85819d54cc7fa57424 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/bc33dfba55f44f85819d54cc7fa57424 2023-03-19 09:56:48,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9fe2909631e74e39b92351dd6caab938 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9fe2909631e74e39b92351dd6caab938 2023-03-19 09:56:48,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/849521f4f5ee4205870e4d9b2bc2f896 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/849521f4f5ee4205870e4d9b2bc2f896 2023-03-19 09:56:48,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0819bad8e6b9447e8a3abab79f5695ca to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/0819bad8e6b9447e8a3abab79f5695ca 2023-03-19 09:56:48,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7b4b41ef1a964af388c7dfab72d1fe85 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/7b4b41ef1a964af388c7dfab72d1fe85 2023-03-19 09:56:48,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/930a23777965496d87c0ec251809f91b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/930a23777965496d87c0ec251809f91b 2023-03-19 09:56:48,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/211e88428635494ab38026a9322b9fac to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/211e88428635494ab38026a9322b9fac 2023-03-19 09:56:48,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/01fb137e97e44246ab5e8063398f3b13 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/01fb137e97e44246ab5e8063398f3b13 2023-03-19 09:56:48,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/92438a70ca2b41d2a075749d70791d19 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/92438a70ca2b41d2a075749d70791d19 2023-03-19 09:56:48,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/e75bf80099d5417a8858e126b17685e5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/e75bf80099d5417a8858e126b17685e5 2023-03-19 09:56:48,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/494c4c14fd8547d18fcac6d872ab23c2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/494c4c14fd8547d18fcac6d872ab23c2 2023-03-19 09:56:48,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b8a445c932c54ca2a57af08007b0901d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b8a445c932c54ca2a57af08007b0901d 2023-03-19 09:56:48,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/78afdfe8fd1146b6a2e9525755bf2668 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/78afdfe8fd1146b6a2e9525755bf2668 2023-03-19 09:56:48,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9e8db2f49fc04fc6ae61332f17360980 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9e8db2f49fc04fc6ae61332f17360980 2023-03-19 09:56:48,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/442024608dc14c0082e06e9dad441dcc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/442024608dc14c0082e06e9dad441dcc 2023-03-19 09:56:48,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/939d3a75ad0a48e98e64751248ff3cb6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/939d3a75ad0a48e98e64751248ff3cb6 2023-03-19 09:56:48,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1e68e70bd04a4fadb9248b1858cb5ba6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/1e68e70bd04a4fadb9248b1858cb5ba6 2023-03-19 09:56:48,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/512d625acee749ecaad837d775b08c4f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/512d625acee749ecaad837d775b08c4f 2023-03-19 09:56:48,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9109fc85e0144fa6be9f5b50fed09c40 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/9109fc85e0144fa6be9f5b50fed09c40 2023-03-19 09:56:48,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/c82f08c1e5d74a0c800ef97acd452702 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/c82f08c1e5d74a0c800ef97acd452702 2023-03-19 09:56:48,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/d73b8a043c6a49d389e1c5d8c6cbefb3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/d73b8a043c6a49d389e1c5d8c6cbefb3 2023-03-19 09:56:48,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/67e82bc6e5fc415086ddfa5a597b935a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/67e82bc6e5fc415086ddfa5a597b935a 2023-03-19 09:56:48,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/956fd2e572864dce86cb7c1bc264d95b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/956fd2e572864dce86cb7c1bc264d95b 2023-03-19 09:56:48,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/14d4323f727c4be5989b896ec5b282df, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/438ad338bc1346aabfb2fe28af0cd870, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6c1f3d1782b04947944803a78db7bfeb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/60922ab72b6c4917b5063009c15c225c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/161c0cdf5921418fbab3daf64660262b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/29f41eba07a34617a0bb6596ef2aa911, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/c62a41dcde59438d85fdc0dc692f11ce, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1788ac20605f4828afa47726aa3f4614, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b773d37920074c9bbd9d355b4c389091, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/71dfd963c2c048edb8e13fe618101e9e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/11e8cbdb1cc940d0990690d49114338c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/39035599cf6c468e9f5c69dd3e49d84a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/82ed10b1018443bfae45525c4853ced2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55618be14551471ba1fa301605961b0c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9ee9dc7a723f4114971474e87385563c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/45e7cb265815401eb7ed03ee244530e3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6ed738c8df2f40d5bb0d4810a4a571ef, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0d5b345c39540ea81a8fd82d0a053e3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e959440e89514f92b4665679b7d98f7e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/ceafb150f8d84b6bbef00bb649970e81, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/03005e846c7f4ef294e418388bbbdade, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d02ab416674046769b5cb5cbe9362ece, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/f63364bed50b46bb80eb61d69417015b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9bd021d3c44647c4bbbe289112ffff8e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/36a14cb330f04b739b8343eb25613da9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/21ab2ab0e688407b8159be2c612f7fc3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b4cde5dc84174eb3992b7dd77c6488d9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9c1bb2cc6e8d4a27afdf5f2ec207274b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e8798ce71fcc40e88159e6716fd9d0d9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7105a550aec44a0b7b88135c2c292fb, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/94578b33fabf412786caf10100dff648, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1b9b4ade783142099e4264672a68920c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1bb34a5060ec429581b7c466d9e67f0c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/300c874367b54934a7d722a956235c98, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/0a74d8b066094bc39278c06766ebe0c3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/af253bcc668f4d0eb35c524d0663127a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/919a9a7f0127484498ff3eca8395c303, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/7cb357bfdb5b4218a58ac0a61cd4a0a6, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55968ac0f4ca4c168f2cac941e9c3414, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/69bdc3365a7f40a781ac6219a4a89d07, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7386f7f31e64291b06d00f7b9149996] to archive 2023-03-19 09:56:48,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:56:48,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/14d4323f727c4be5989b896ec5b282df to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/14d4323f727c4be5989b896ec5b282df 2023-03-19 09:56:48,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/438ad338bc1346aabfb2fe28af0cd870 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/438ad338bc1346aabfb2fe28af0cd870 2023-03-19 09:56:48,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6c1f3d1782b04947944803a78db7bfeb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6c1f3d1782b04947944803a78db7bfeb 2023-03-19 09:56:48,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/60922ab72b6c4917b5063009c15c225c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/60922ab72b6c4917b5063009c15c225c 2023-03-19 09:56:48,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/161c0cdf5921418fbab3daf64660262b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/161c0cdf5921418fbab3daf64660262b 2023-03-19 09:56:48,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/29f41eba07a34617a0bb6596ef2aa911 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/29f41eba07a34617a0bb6596ef2aa911 2023-03-19 09:56:48,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/c62a41dcde59438d85fdc0dc692f11ce to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/c62a41dcde59438d85fdc0dc692f11ce 2023-03-19 09:56:48,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1788ac20605f4828afa47726aa3f4614 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1788ac20605f4828afa47726aa3f4614 2023-03-19 09:56:48,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b773d37920074c9bbd9d355b4c389091 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b773d37920074c9bbd9d355b4c389091 2023-03-19 09:56:48,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/71dfd963c2c048edb8e13fe618101e9e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/71dfd963c2c048edb8e13fe618101e9e 2023-03-19 09:56:48,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/11e8cbdb1cc940d0990690d49114338c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/11e8cbdb1cc940d0990690d49114338c 2023-03-19 09:56:48,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/39035599cf6c468e9f5c69dd3e49d84a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/39035599cf6c468e9f5c69dd3e49d84a 2023-03-19 09:56:48,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/82ed10b1018443bfae45525c4853ced2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/82ed10b1018443bfae45525c4853ced2 2023-03-19 09:56:48,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55618be14551471ba1fa301605961b0c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55618be14551471ba1fa301605961b0c 2023-03-19 09:56:48,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9ee9dc7a723f4114971474e87385563c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9ee9dc7a723f4114971474e87385563c 2023-03-19 09:56:48,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/45e7cb265815401eb7ed03ee244530e3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/45e7cb265815401eb7ed03ee244530e3 2023-03-19 09:56:48,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6ed738c8df2f40d5bb0d4810a4a571ef to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/6ed738c8df2f40d5bb0d4810a4a571ef 2023-03-19 09:56:48,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0d5b345c39540ea81a8fd82d0a053e3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0d5b345c39540ea81a8fd82d0a053e3 2023-03-19 09:56:48,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e959440e89514f92b4665679b7d98f7e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e959440e89514f92b4665679b7d98f7e 2023-03-19 09:56:48,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/ceafb150f8d84b6bbef00bb649970e81 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/ceafb150f8d84b6bbef00bb649970e81 2023-03-19 09:56:48,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/03005e846c7f4ef294e418388bbbdade to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/03005e846c7f4ef294e418388bbbdade 2023-03-19 09:56:48,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d02ab416674046769b5cb5cbe9362ece to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d02ab416674046769b5cb5cbe9362ece 2023-03-19 09:56:48,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/f63364bed50b46bb80eb61d69417015b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/f63364bed50b46bb80eb61d69417015b 2023-03-19 09:56:48,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9bd021d3c44647c4bbbe289112ffff8e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9bd021d3c44647c4bbbe289112ffff8e 2023-03-19 09:56:48,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/36a14cb330f04b739b8343eb25613da9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/36a14cb330f04b739b8343eb25613da9 2023-03-19 09:56:48,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/21ab2ab0e688407b8159be2c612f7fc3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/21ab2ab0e688407b8159be2c612f7fc3 2023-03-19 09:56:48,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b4cde5dc84174eb3992b7dd77c6488d9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/b4cde5dc84174eb3992b7dd77c6488d9 2023-03-19 09:56:48,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/fc8ec5e7f4d84e47ab0a5ad96fc7bda5 2023-03-19 09:56:48,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9c1bb2cc6e8d4a27afdf5f2ec207274b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/9c1bb2cc6e8d4a27afdf5f2ec207274b 2023-03-19 09:56:48,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e8798ce71fcc40e88159e6716fd9d0d9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e8798ce71fcc40e88159e6716fd9d0d9 2023-03-19 09:56:48,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7105a550aec44a0b7b88135c2c292fb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7105a550aec44a0b7b88135c2c292fb 2023-03-19 09:56:48,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/94578b33fabf412786caf10100dff648 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/94578b33fabf412786caf10100dff648 2023-03-19 09:56:48,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1b9b4ade783142099e4264672a68920c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1b9b4ade783142099e4264672a68920c 2023-03-19 09:56:48,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1bb34a5060ec429581b7c466d9e67f0c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/1bb34a5060ec429581b7c466d9e67f0c 2023-03-19 09:56:48,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/300c874367b54934a7d722a956235c98 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/300c874367b54934a7d722a956235c98 2023-03-19 09:56:48,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/0a74d8b066094bc39278c06766ebe0c3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/0a74d8b066094bc39278c06766ebe0c3 2023-03-19 09:56:48,815 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/af253bcc668f4d0eb35c524d0663127a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/af253bcc668f4d0eb35c524d0663127a 2023-03-19 09:56:48,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/919a9a7f0127484498ff3eca8395c303 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/919a9a7f0127484498ff3eca8395c303 2023-03-19 09:56:48,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/7cb357bfdb5b4218a58ac0a61cd4a0a6 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/7cb357bfdb5b4218a58ac0a61cd4a0a6 2023-03-19 09:56:48,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55968ac0f4ca4c168f2cac941e9c3414 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/55968ac0f4ca4c168f2cac941e9c3414 2023-03-19 09:56:48,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/69bdc3365a7f40a781ac6219a4a89d07 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/69bdc3365a7f40a781ac6219a4a89d07 2023-03-19 09:56:48,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7386f7f31e64291b06d00f7b9149996 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/d7386f7f31e64291b06d00f7b9149996 2023-03-19 09:56:48,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fd16037091d446ff9a15f4a82219210c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/6b70e918ac7245c19f86a5a69a4d70ad, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/becc227127f24f6b825701f8bbca7053, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/c4143803a85d47e7b16a94b8e941bcbf, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fcb3f01aacac4eeda2e5a1b0fd83d1ba, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/2dd953283ca14521a15aeb074f774b54, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/eafc34d011914c919ab0a7652eba7556, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a13102c9479f44448eb02e0674e27416, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3531f1c7e87545029d43fccc67fcca46, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/965513e5d0c342d5a833ef73080e4262, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/988b7be44fd2405aa3238e120342b0a1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b8d84b19a94e4a3ca5e0cb90b44ae264, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/daea4e44f5954268950d6c0e2842ecd3, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cc007ca695224e189968bd2c9a8147b1, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b485aaca876342e881842a142299e526, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/4f3326e9b69a4dbab70b0f1b92bbe189, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/448bc40bc7f049ebbddd6b718239e50e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/5ad658edaa10461aa9b260c04c76ca05, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/459d53b87c714a01958b0067177c09f2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0041e98ddae341bab7e8480d961ce41b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b039183288404041967e37e17447b92c, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b499e1a8922a457296899f6325a70d48, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1d674337aa324eacb4f52765a216094d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09b5db4d7a4d411dbf1328c550a408aa, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/af3c9dc380f44090acb20482ca6205c2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/33bb3ac00a504444bd6eda6b30d005f5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9b28c17a00ea44cf8c0c5177bc8d5c8f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a4dd03d584a847538ab55f27379d0c1e, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1bdf55b990eb405f9c103655b8886cc5, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/8fa0725e2a494f078c6a728300db3789, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/13a0cd3db77f4b5fbb717f2f00711e4b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09d4e10b4ce04c5592c12e8d2c5bb19f, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b0e2ddd402ea4eb8b62207fff78d9611, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9cb474b3286e4330a4a7eaec8fa9cadc, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cffe47cc9a5042a3b36d2d94a112ebb9, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/567b81beb35b4aaa90d237f2f35036db, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/53e47021cf734091807b7218e214fc1b, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a2fcbab69a5a42328ab74ca0ca334da2, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/130a18271d164ddeb486691a7b54035a, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0deec4409bc945d0a99485efb16d312d, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3d949a6fa2ed466bb9b49f01189a03ac, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/97dfcd691d95452190c485aa9215aea2] to archive 2023-03-19 09:56:48,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-19 09:56:48,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fd16037091d446ff9a15f4a82219210c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fd16037091d446ff9a15f4a82219210c 2023-03-19 09:56:48,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/6b70e918ac7245c19f86a5a69a4d70ad to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/6b70e918ac7245c19f86a5a69a4d70ad 2023-03-19 09:56:48,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/becc227127f24f6b825701f8bbca7053 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/becc227127f24f6b825701f8bbca7053 2023-03-19 09:56:48,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/c4143803a85d47e7b16a94b8e941bcbf to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/c4143803a85d47e7b16a94b8e941bcbf 2023-03-19 09:56:48,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fcb3f01aacac4eeda2e5a1b0fd83d1ba to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/fcb3f01aacac4eeda2e5a1b0fd83d1ba 2023-03-19 09:56:48,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/2dd953283ca14521a15aeb074f774b54 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/2dd953283ca14521a15aeb074f774b54 2023-03-19 09:56:48,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/eafc34d011914c919ab0a7652eba7556 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/eafc34d011914c919ab0a7652eba7556 2023-03-19 09:56:48,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a13102c9479f44448eb02e0674e27416 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a13102c9479f44448eb02e0674e27416 2023-03-19 09:56:48,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3531f1c7e87545029d43fccc67fcca46 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3531f1c7e87545029d43fccc67fcca46 2023-03-19 09:56:48,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/965513e5d0c342d5a833ef73080e4262 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/965513e5d0c342d5a833ef73080e4262 2023-03-19 09:56:48,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/988b7be44fd2405aa3238e120342b0a1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/988b7be44fd2405aa3238e120342b0a1 2023-03-19 09:56:48,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b8d84b19a94e4a3ca5e0cb90b44ae264 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b8d84b19a94e4a3ca5e0cb90b44ae264 2023-03-19 09:56:48,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/daea4e44f5954268950d6c0e2842ecd3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/daea4e44f5954268950d6c0e2842ecd3 2023-03-19 09:56:48,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cc007ca695224e189968bd2c9a8147b1 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cc007ca695224e189968bd2c9a8147b1 2023-03-19 09:56:48,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b485aaca876342e881842a142299e526 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b485aaca876342e881842a142299e526 2023-03-19 09:56:48,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/4f3326e9b69a4dbab70b0f1b92bbe189 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/4f3326e9b69a4dbab70b0f1b92bbe189 2023-03-19 09:56:48,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/448bc40bc7f049ebbddd6b718239e50e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/448bc40bc7f049ebbddd6b718239e50e 2023-03-19 09:56:48,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/5ad658edaa10461aa9b260c04c76ca05 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/5ad658edaa10461aa9b260c04c76ca05 2023-03-19 09:56:48,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/459d53b87c714a01958b0067177c09f2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/459d53b87c714a01958b0067177c09f2 2023-03-19 09:56:48,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0041e98ddae341bab7e8480d961ce41b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0041e98ddae341bab7e8480d961ce41b 2023-03-19 09:56:48,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b039183288404041967e37e17447b92c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b039183288404041967e37e17447b92c 2023-03-19 09:56:48,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b499e1a8922a457296899f6325a70d48 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b499e1a8922a457296899f6325a70d48 2023-03-19 09:56:48,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1d674337aa324eacb4f52765a216094d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1d674337aa324eacb4f52765a216094d 2023-03-19 09:56:48,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09b5db4d7a4d411dbf1328c550a408aa to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09b5db4d7a4d411dbf1328c550a408aa 2023-03-19 09:56:48,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/af3c9dc380f44090acb20482ca6205c2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/af3c9dc380f44090acb20482ca6205c2 2023-03-19 09:56:48,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/33bb3ac00a504444bd6eda6b30d005f5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/33bb3ac00a504444bd6eda6b30d005f5 2023-03-19 09:56:48,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9b28c17a00ea44cf8c0c5177bc8d5c8f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9b28c17a00ea44cf8c0c5177bc8d5c8f 2023-03-19 09:56:48,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a4dd03d584a847538ab55f27379d0c1e to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a4dd03d584a847538ab55f27379d0c1e 2023-03-19 09:56:48,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1bdf55b990eb405f9c103655b8886cc5 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/1bdf55b990eb405f9c103655b8886cc5 2023-03-19 09:56:48,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/8fa0725e2a494f078c6a728300db3789 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/8fa0725e2a494f078c6a728300db3789 2023-03-19 09:56:48,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/13a0cd3db77f4b5fbb717f2f00711e4b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/13a0cd3db77f4b5fbb717f2f00711e4b 2023-03-19 09:56:48,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09d4e10b4ce04c5592c12e8d2c5bb19f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/09d4e10b4ce04c5592c12e8d2c5bb19f 2023-03-19 09:56:48,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b0e2ddd402ea4eb8b62207fff78d9611 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/b0e2ddd402ea4eb8b62207fff78d9611 2023-03-19 09:56:48,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9cb474b3286e4330a4a7eaec8fa9cadc to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/9cb474b3286e4330a4a7eaec8fa9cadc 2023-03-19 09:56:48,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cffe47cc9a5042a3b36d2d94a112ebb9 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/cffe47cc9a5042a3b36d2d94a112ebb9 2023-03-19 09:56:48,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/567b81beb35b4aaa90d237f2f35036db to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/567b81beb35b4aaa90d237f2f35036db 2023-03-19 09:56:48,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/53e47021cf734091807b7218e214fc1b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/53e47021cf734091807b7218e214fc1b 2023-03-19 09:56:48,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a2fcbab69a5a42328ab74ca0ca334da2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/a2fcbab69a5a42328ab74ca0ca334da2 2023-03-19 09:56:48,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/130a18271d164ddeb486691a7b54035a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/130a18271d164ddeb486691a7b54035a 2023-03-19 09:56:48,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0deec4409bc945d0a99485efb16d312d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/0deec4409bc945d0a99485efb16d312d 2023-03-19 09:56:48,856 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3d949a6fa2ed466bb9b49f01189a03ac to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/3d949a6fa2ed466bb9b49f01189a03ac 2023-03-19 09:56:48,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/97dfcd691d95452190c485aa9215aea2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/97dfcd691d95452190c485aa9215aea2 2023-03-19 09:56:48,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/recovered.edits/1021.seqid, newMaxSeqId=1021, maxSeqId=4 2023-03-19 09:56:48,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901. 2023-03-19 09:56:48,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a391bedbca383dedc4bc8fc94dd92901: 2023-03-19 09:56:48,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:48,862 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a391bedbca383dedc4bc8fc94dd92901, regionState=CLOSED 2023-03-19 09:56:48,863 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679219808862"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679219808862"}]},"ts":"1679219808862"} 2023-03-19 09:56:48,864 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-19 09:56:48,864 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure a391bedbca383dedc4bc8fc94dd92901, server=jenkins-hbase4.apache.org,34669,1679219618625 in 3.0410 sec 2023-03-19 09:56:48,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-19 09:56:48,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a391bedbca383dedc4bc8fc94dd92901, UNASSIGN in 3.0450 sec 2023-03-19 09:56:48,866 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679219808866"}]},"ts":"1679219808866"} 2023-03-19 09:56:48,866 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-19 09:56:48,869 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-19 09:56:48,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.0550 sec 2023-03-19 09:56:49,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-19 09:56:49,922 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-19 09:56:49,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-19 09:56:49,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:49,924 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:49,925 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:49,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-19 09:56:49,930 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:49,932 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C, FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/recovered.edits] 2023-03-19 09:56:49,935 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b15e7bda18824097b401f5fc0b877c51 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/b15e7bda18824097b401f5fc0b877c51 2023-03-19 09:56:49,936 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/f383e0d1124349aeae3db6665997139f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/A/f383e0d1124349aeae3db6665997139f 2023-03-19 09:56:49,938 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0091e3029244976b1b8b4689dc3a1a2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e0091e3029244976b1b8b4689dc3a1a2 2023-03-19 09:56:49,939 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e7197c6e5d21452c812cf10e328d7993 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/B/e7197c6e5d21452c812cf10e328d7993 2023-03-19 09:56:49,941 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/20433e82fc8f4ccb96c385128a4a636f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/20433e82fc8f4ccb96c385128a4a636f 2023-03-19 09:56:49,942 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/7e6213f439b34c8f951e6693af436df7 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/C/7e6213f439b34c8f951e6693af436df7 2023-03-19 09:56:49,944 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/recovered.edits/1021.seqid to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901/recovered.edits/1021.seqid 2023-03-19 09:56:49,944 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/.tmp/data/default/TestAcidGuarantees/a391bedbca383dedc4bc8fc94dd92901 2023-03-19 09:56:49,944 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-19 09:56:49,945 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-19 09:56:49,945 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-19 09:56:49,947 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031901645ed2078e4131a2b50ecff6a05406 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031901645ed2078e4131a2b50ecff6a05406 2023-03-19 09:56:49,948 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319145df27b07004cb4a10efa9c2deaceb8 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319145df27b07004cb4a10efa9c2deaceb8 2023-03-19 09:56:49,949 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303192808b5c6f3a6402883c322c6cff7cc72 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303192808b5c6f3a6402883c322c6cff7cc72 2023-03-19 09:56:49,950 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193091bfee825f42128dacf2e6d1a9a917 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193091bfee825f42128dacf2e6d1a9a917 2023-03-19 09:56:49,951 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031936dd5bd0d7da489ab523a6416ceee0ea to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031936dd5bd0d7da489ab523a6416ceee0ea 2023-03-19 09:56:49,952 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193818d7e0683d46d8b6a351acaac27ce2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193818d7e0683d46d8b6a351acaac27ce2 2023-03-19 09:56:49,953 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031939dc9d7a05c04ada88810b383d94a275 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031939dc9d7a05c04ada88810b383d94a275 2023-03-19 09:56:49,954 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193b15110eeaca47ff92cb6374c47fd4da to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193b15110eeaca47ff92cb6374c47fd4da 2023-03-19 09:56:49,955 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193b76974f52cd401883429b1f0ec36c84 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193b76974f52cd401883429b1f0ec36c84 2023-03-19 09:56:49,955 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193ee3684975ba429687948197d78c513b to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303193ee3684975ba429687948197d78c513b 2023-03-19 09:56:49,956 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031946f10a5cc07c4fad9021f3b30a1f56bb to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031946f10a5cc07c4fad9021f3b30a1f56bb 2023-03-19 09:56:49,958 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319493516085d914f0b9765047e709f0cfe to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319493516085d914f0b9765047e709f0cfe 2023-03-19 09:56:49,959 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195c4686fb6d4b4cd683cbc6e1c3afd590 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303195c4686fb6d4b4cd683cbc6e1c3afd590 2023-03-19 09:56:49,960 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196c1218359c6f4e03b5210a15bb71eade to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196c1218359c6f4e03b5210a15bb71eade 2023-03-19 09:56:49,961 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196ce5cc152a7249389739e3bcbf745f64 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303196ce5cc152a7249389739e3bcbf745f64 2023-03-19 09:56:49,962 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197bbe0189505d469ab4d92dc15d85d021 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303197bbe0189505d469ab4d92dc15d85d021 2023-03-19 09:56:49,963 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198d4c87019b544a7dadacc86b3428be3a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303198d4c87019b544a7dadacc86b3428be3a 2023-03-19 09:56:49,965 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031993d370f794104befaab554b4602edd52 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023031993d370f794104befaab554b4602edd52 2023-03-19 09:56:49,966 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a481b4c3eaec4ce49df35b55d60e2d7f to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a481b4c3eaec4ce49df35b55d60e2d7f 2023-03-19 09:56:49,967 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a53e5c942ef54a3aad08f35699194a92 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a53e5c942ef54a3aad08f35699194a92 2023-03-19 09:56:49,968 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a5a9dad56f0e41e79c23962efdf03b87 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a5a9dad56f0e41e79c23962efdf03b87 2023-03-19 09:56:49,970 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a8bd13bbab0c4eb897d76810c0056876 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319a8bd13bbab0c4eb897d76810c0056876 2023-03-19 09:56:49,971 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319aadd75c552f245e1b743fa974a136d8c to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319aadd75c552f245e1b743fa974a136d8c 2023-03-19 09:56:49,972 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ace95c0e362c48ff9701c1de4401dfd3 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ace95c0e362c48ff9701c1de4401dfd3 2023-03-19 09:56:49,973 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b7888e5698774b0fbe92a413788be339 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319b7888e5698774b0fbe92a413788be339 2023-03-19 09:56:49,974 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ba5e31cd99aa4a2c955cd2ceb76b33a2 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ba5e31cd99aa4a2c955cd2ceb76b33a2 2023-03-19 09:56:49,975 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319be500c48dfc747ef90c9da8629186bcd to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319be500c48dfc747ef90c9da8629186bcd 2023-03-19 09:56:49,976 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319cd61854b80ec4da9b070dc368f535845 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319cd61854b80ec4da9b070dc368f535845 2023-03-19 09:56:49,977 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ce645a2876ba45e395f0488faf0cf942 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ce645a2876ba45e395f0488faf0cf942 2023-03-19 09:56:49,980 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319daf794f3aaa843b499a00e36d7df6549 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319daf794f3aaa843b499a00e36d7df6549 2023-03-19 09:56:49,981 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ea388d72621941078745c74836c2b22a to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ea388d72621941078745c74836c2b22a 2023-03-19 09:56:49,982 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ed3deccbb4c94d0dbf1cd1fcdd8d8b30 to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319ed3deccbb4c94d0dbf1cd1fcdd8d8b30 2023-03-19 09:56:49,983 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319f7c83ae8742741e4829187ec92c4612d to hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230319f7c83ae8742741e4829187ec92c4612d 2023-03-19 09:56:49,984 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-19 09:56:49,986 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:49,987 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-19 09:56:49,989 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-19 09:56:49,990 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:49,990 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-19 09:56:49,990 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679219809990"}]},"ts":"9223372036854775807"} 2023-03-19 09:56:49,991 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-19 09:56:49,991 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a391bedbca383dedc4bc8fc94dd92901, NAME => 'TestAcidGuarantees,,1679219779096.a391bedbca383dedc4bc8fc94dd92901.', STARTKEY => '', ENDKEY => ''}] 2023-03-19 09:56:49,991 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-19 09:56:49,992 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679219809992"}]},"ts":"9223372036854775807"} 2023-03-19 09:56:49,993 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-19 09:56:49,995 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-19 09:56:49,996 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 73 msec 2023-03-19 09:56:50,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44865] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-19 09:56:50,026 INFO [Listener at localhost/45961] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-19 09:56:50,038 INFO [Listener at localhost/45961] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=226 (was 225) - Thread LEAK? -, OpenFileDescriptor=619 (was 524) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=383 (was 342) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1736 (was 1738) 2023-03-19 09:56:50,039 INFO [Listener at localhost/45961] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-19 09:56:50,039 INFO [Listener at localhost/45961] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-19 09:56:50,039 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:54245 2023-03-19 09:56:50,039 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:50,039 DEBUG [Listener at localhost/45961] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-19 09:56:50,039 DEBUG [Listener at localhost/45961] util.JVMClusterUtil(257): Found active master hash=757396817, stopped=false 2023-03-19 09:56:50,039 INFO [Listener at localhost/45961] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:56:50,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-19 09:56:50,040 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-19 09:56:50,040 INFO [Listener at localhost/45961] procedure2.ProcedureExecutor(629): Stopping 2023-03-19 09:56:50,041 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:56:50,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-19 09:56:50,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-19 09:56:50,041 DEBUG [Listener at localhost/45961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a5d5fe6 to 127.0.0.1:54245 2023-03-19 09:56:50,041 DEBUG [Listener at localhost/45961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:50,041 INFO [Listener at localhost/45961] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34669,1679219618625' ***** 2023-03-19 09:56:50,041 INFO [Listener at localhost/45961] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-19 09:56:50,042 INFO [RS:0;jenkins-hbase4:34669] regionserver.HeapMemoryManager(220): Stopping 2023-03-19 09:56:50,042 INFO [RS:0;jenkins-hbase4:34669] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-19 09:56:50,042 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-19 09:56:50,042 INFO [RS:0;jenkins-hbase4:34669] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-19 09:56:50,042 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(3304): Received CLOSE for c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:56:50,043 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:50,043 DEBUG [RS:0;jenkins-hbase4:34669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x643192bc to 127.0.0.1:54245 2023-03-19 09:56:50,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c405ddb2855a0ca7512046e9ce410120, disabling compactions & flushes 2023-03-19 09:56:50,044 DEBUG [RS:0;jenkins-hbase4:34669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:50,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:56:50,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:56:50,044 INFO [RS:0;jenkins-hbase4:34669] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-19 09:56:50,044 INFO [RS:0;jenkins-hbase4:34669] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-19 09:56:50,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. after waiting 0 ms 2023-03-19 09:56:50,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:56:50,044 INFO [RS:0;jenkins-hbase4:34669] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-19 09:56:50,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c405ddb2855a0ca7512046e9ce410120 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-19 09:56:50,045 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-19 09:56:50,045 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-19 09:56:50,046 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, c405ddb2855a0ca7512046e9ce410120=hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120.} 2023-03-19 09:56:50,047 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1505): Waiting on 1588230740, c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:56:50,047 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-19 09:56:50,048 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-19 09:56:50,048 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-19 09:56:50,048 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-19 09:56:50,048 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-19 09:56:50,048 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-19 09:56:50,058 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/.tmp/info/aa609bcb830e458485f5a924330b1152 2023-03-19 09:56:50,062 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aa609bcb830e458485f5a924330b1152 2023-03-19 09:56:50,067 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/.tmp/rep_barrier/239105d0abb84d2c955d0825c22fe999 2023-03-19 09:56:50,072 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 239105d0abb84d2c955d0825c22fe999 2023-03-19 09:56:50,081 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/.tmp/table/c3e69913a04f4cd4ba96f3bf5e78b7cb 2023-03-19 09:56:50,085 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c3e69913a04f4cd4ba96f3bf5e78b7cb 2023-03-19 09:56:50,085 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/.tmp/info/aa609bcb830e458485f5a924330b1152 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/info/aa609bcb830e458485f5a924330b1152 2023-03-19 09:56:50,088 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aa609bcb830e458485f5a924330b1152 2023-03-19 09:56:50,089 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/info/aa609bcb830e458485f5a924330b1152, entries=22, sequenceid=93, filesize=7.1 K 2023-03-19 09:56:50,089 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/.tmp/rep_barrier/239105d0abb84d2c955d0825c22fe999 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/rep_barrier/239105d0abb84d2c955d0825c22fe999 2023-03-19 09:56:50,093 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 239105d0abb84d2c955d0825c22fe999 2023-03-19 09:56:50,093 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/rep_barrier/239105d0abb84d2c955d0825c22fe999, entries=6, sequenceid=93, filesize=5.5 K 2023-03-19 09:56:50,093 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/.tmp/table/c3e69913a04f4cd4ba96f3bf5e78b7cb as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/table/c3e69913a04f4cd4ba96f3bf5e78b7cb 2023-03-19 09:56:50,096 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-19 09:56:50,097 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c3e69913a04f4cd4ba96f3bf5e78b7cb 2023-03-19 09:56:50,097 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/table/c3e69913a04f4cd4ba96f3bf5e78b7cb, entries=9, sequenceid=93, filesize=5.5 K 2023-03-19 09:56:50,097 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 49ms, sequenceid=93, compaction requested=false 2023-03-19 09:56:50,105 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-19 09:56:50,106 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-19 09:56:50,107 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-19 09:56:50,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-19 09:56:50,107 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-19 09:56:50,248 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1505): Waiting on c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:56:50,448 DEBUG [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1505): Waiting on c405ddb2855a0ca7512046e9ce410120 2023-03-19 09:56:50,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/.tmp/info/779360affff94aec9a4997507875793f 2023-03-19 09:56:50,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/.tmp/info/779360affff94aec9a4997507875793f as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/info/779360affff94aec9a4997507875793f 2023-03-19 09:56:50,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/info/779360affff94aec9a4997507875793f, entries=2, sequenceid=6, filesize=4.8 K 2023-03-19 09:56:50,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c405ddb2855a0ca7512046e9ce410120 in 421ms, sequenceid=6, compaction requested=false 2023-03-19 09:56:50,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/data/hbase/namespace/c405ddb2855a0ca7512046e9ce410120/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-19 09:56:50,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:56:50,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c405ddb2855a0ca7512046e9ce410120: 2023-03-19 09:56:50,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679219620497.c405ddb2855a0ca7512046e9ce410120. 2023-03-19 09:56:50,648 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34669,1679219618625; all regions closed. 2023-03-19 09:56:50,655 DEBUG [RS:0;jenkins-hbase4:34669] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/oldWALs 2023-03-19 09:56:50,655 INFO [RS:0;jenkins-hbase4:34669] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34669%2C1679219618625.meta:.meta(num 1679219620227) 2023-03-19 09:56:50,659 DEBUG [RS:0;jenkins-hbase4:34669] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/oldWALs 2023-03-19 09:56:50,659 INFO [RS:0;jenkins-hbase4:34669] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34669%2C1679219618625:(num 1679219620011) 2023-03-19 09:56:50,659 DEBUG [RS:0;jenkins-hbase4:34669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:50,659 INFO [RS:0;jenkins-hbase4:34669] regionserver.LeaseManager(133): Closed leases 2023-03-19 09:56:50,659 INFO [RS:0;jenkins-hbase4:34669] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-19 09:56:50,660 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-19 09:56:50,661 INFO [RS:0;jenkins-hbase4:34669] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34669 2023-03-19 09:56:50,666 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34669,1679219618625 2023-03-19 09:56:50,666 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-19 09:56:50,666 ERROR [Listener at localhost/45961-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2d2e1e65 rejected from java.util.concurrent.ThreadPoolExecutor@29118f18[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2719] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-19 09:56:50,667 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-19 09:56:50,667 ERROR [Listener at localhost/45961-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b56710 rejected from java.util.concurrent.ThreadPoolExecutor@29118f18[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2719] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-19 09:56:50,670 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34669,1679219618625] 2023-03-19 09:56:50,670 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34669,1679219618625; numProcessing=1 2023-03-19 09:56:50,671 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34669,1679219618625 already deleted, retry=false 2023-03-19 09:56:50,671 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34669,1679219618625 expired; onlineServers=0 2023-03-19 09:56:50,671 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44865,1679219617257' ***** 2023-03-19 09:56:50,671 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-19 09:56:50,672 DEBUG [M:0;jenkins-hbase4:44865] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c19ff6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-19 09:56:50,672 INFO [M:0;jenkins-hbase4:44865] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44865,1679219617257 2023-03-19 09:56:50,672 INFO [M:0;jenkins-hbase4:44865] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44865,1679219617257; all regions closed. 2023-03-19 09:56:50,672 DEBUG [M:0;jenkins-hbase4:44865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-19 09:56:50,672 DEBUG [M:0;jenkins-hbase4:44865] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-19 09:56:50,672 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-19 09:56:50,672 DEBUG [M:0;jenkins-hbase4:44865] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-19 09:56:50,672 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679219619710] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679219619710,5,FailOnTimeoutGroup] 2023-03-19 09:56:50,672 INFO [M:0;jenkins-hbase4:44865] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-19 09:56:50,672 INFO [M:0;jenkins-hbase4:44865] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-19 09:56:50,672 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679219619711] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679219619711,5,FailOnTimeoutGroup] 2023-03-19 09:56:50,672 INFO [M:0;jenkins-hbase4:44865] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-19 09:56:50,673 DEBUG [M:0;jenkins-hbase4:44865] master.HMaster(1512): Stopping service threads 2023-03-19 09:56:50,673 INFO [M:0;jenkins-hbase4:44865] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-19 09:56:50,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-19 09:56:50,673 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-19 09:56:50,673 ERROR [M:0;jenkins-hbase4:44865] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (341669413) connection to localhost/127.0.0.1:39255 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-19 09:56:50,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-19 09:56:50,674 INFO [M:0;jenkins-hbase4:44865] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-19 09:56:50,674 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-19 09:56:50,674 DEBUG [M:0;jenkins-hbase4:44865] zookeeper.ZKUtil(398): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-19 09:56:50,674 WARN [M:0;jenkins-hbase4:44865] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-19 09:56:50,674 INFO [M:0;jenkins-hbase4:44865] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-19 09:56:50,674 INFO [M:0;jenkins-hbase4:44865] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-19 09:56:50,675 DEBUG [M:0;jenkins-hbase4:44865] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-19 09:56:50,675 INFO [M:0;jenkins-hbase4:44865] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:56:50,675 DEBUG [M:0;jenkins-hbase4:44865] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:56:50,675 DEBUG [M:0;jenkins-hbase4:44865] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-19 09:56:50,675 DEBUG [M:0;jenkins-hbase4:44865] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:56:50,675 INFO [M:0;jenkins-hbase4:44865] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.85 KB heapSize=393.55 KB 2023-03-19 09:56:50,686 INFO [M:0;jenkins-hbase4:44865] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.85 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fa1cf4641254423b9f643d3c329c4eb0 2023-03-19 09:56:50,690 INFO [M:0;jenkins-hbase4:44865] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fa1cf4641254423b9f643d3c329c4eb0 2023-03-19 09:56:50,690 DEBUG [M:0;jenkins-hbase4:44865] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fa1cf4641254423b9f643d3c329c4eb0 as hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fa1cf4641254423b9f643d3c329c4eb0 2023-03-19 09:56:50,694 INFO [M:0;jenkins-hbase4:44865] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fa1cf4641254423b9f643d3c329c4eb0 2023-03-19 09:56:50,694 INFO [M:0;jenkins-hbase4:44865] regionserver.HStore(1080): Added hdfs://localhost:39255/user/jenkins/test-data/50228f97-7235-eba4-a49b-09b203071505/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fa1cf4641254423b9f643d3c329c4eb0, entries=65, sequenceid=578, filesize=24.7 K 2023-03-19 09:56:50,694 INFO [M:0;jenkins-hbase4:44865] regionserver.HRegion(2948): Finished flush of dataSize ~344.85 KB/353123, heapSize ~393.54 KB/402984, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=578, compaction requested=false 2023-03-19 09:56:50,696 INFO [M:0;jenkins-hbase4:44865] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-19 09:56:50,696 DEBUG [M:0;jenkins-hbase4:44865] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-19 09:56:50,698 INFO [M:0;jenkins-hbase4:44865] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-19 09:56:50,698 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-19 09:56:50,699 INFO [M:0;jenkins-hbase4:44865] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44865 2023-03-19 09:56:50,700 DEBUG [M:0;jenkins-hbase4:44865] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,44865,1679219617257 already deleted, retry=false 2023-03-19 09:56:50,771 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-19 09:56:50,771 INFO [RS:0;jenkins-hbase4:34669] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34669,1679219618625; zookeeper connection closed. 2023-03-19 09:56:50,771 ERROR [Listener at localhost/45961-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@78f9b1b4 rejected from java.util.concurrent.ThreadPoolExecutor@29118f18[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2719] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-19 09:56:50,771 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@19b9cd0a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@19b9cd0a 2023-03-19 09:56:50,771 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): regionserver:34669-0x100fc31d44a0001, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-19 09:56:50,772 ERROR [Listener at localhost/45961-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@279a8061 rejected from java.util.concurrent.ThreadPoolExecutor@29118f18[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2719] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-19 09:56:50,772 INFO [Listener at localhost/45961] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-19 09:56:50,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-19 09:56:50,871 INFO [M:0;jenkins-hbase4:44865] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44865,1679219617257; zookeeper connection closed. 2023-03-19 09:56:50,871 ERROR [Listener at localhost/45961-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@151f7b65 rejected from java.util.concurrent.ThreadPoolExecutor@6a4b899f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4541] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-19 09:56:50,871 DEBUG [Listener at localhost/45961-EventThread] zookeeper.ZKWatcher(600): master:44865-0x100fc31d44a0000, quorum=127.0.0.1:54245, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-19 09:56:50,871 ERROR [Listener at localhost/45961-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@29d48cdd rejected from java.util.concurrent.ThreadPoolExecutor@6a4b899f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4541] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-19 09:56:50,873 WARN [Listener at localhost/45961] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-19 09:56:50,876 INFO [Listener at localhost/45961] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-19 09:56:50,984 WARN [BP-173179777-172.31.14.131-1679219613898 heartbeating to localhost/127.0.0.1:39255] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-19 09:56:50,985 WARN [BP-173179777-172.31.14.131-1679219613898 heartbeating to localhost/127.0.0.1:39255] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-173179777-172.31.14.131-1679219613898 (Datanode Uuid 8ae7b5f6-b356-4e6a-a1cd-cdb73824920a) service to localhost/127.0.0.1:39255 2023-03-19 09:56:50,987 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/cluster_7a353873-d325-433e-e096-98958ab2209b/dfs/data/data1/current/BP-173179777-172.31.14.131-1679219613898] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-19 09:56:50,987 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3fa98f05-97cb-1ae6-0f0f-f8134180c427/cluster_7a353873-d325-433e-e096-98958ab2209b/dfs/data/data2/current/BP-173179777-172.31.14.131-1679219613898] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-19 09:56:51,021 INFO [Listener at localhost/45961] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-19 09:56:51,193 INFO [Listener at localhost/45961] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-19 09:56:51,385 INFO [Listener at localhost/45961] hbase.HBaseTestingUtility(1293): Minicluster is down